[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:07:46 dist-test-slave-2pm7 krb5kdc[469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:07:46 dist-test-slave-2pm7 krb5kdc[469](info): set up 2 sockets
Sep 24 20:07:46 dist-test-slave-2pm7 krb5kdc[469](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 20:07:47.612978 462 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:07:47 dist-test-slave-2pm7 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:07:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:07:47.744081 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33915
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:47.854283 482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:47.854591 482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:47.854666 482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:47.858644 482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:07:47.858734 482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:47.858773 482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:47.858810 482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:07:47.858846 482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:07:47.864717 482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:33915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:47.865860 482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:47.866916 482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:47.876137 490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:47.876152 487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:47.876152 488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:47.876515 482 server_base.cc:1034] running on GCE node
I20260924 20:07:47.877187 482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:47.878412 482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:47.879622 482 hybrid_clock.cc:648] HybridClock initialized: now 1790280467879588 us; error 44 us; skew 500 ppm
Sep 24 20:07:47 dist-test-slave-2pm7 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:47.882476 482 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:07:47.883416 482 webserver.cc:460] Webserver started at http://127.0.115.190:41601/ using document root <none> and password file <none>
I20260924 20:07:47.884081 482 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:47.884181 482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:47.884410 482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:47.886457 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "e39d82fd528e4099a7f59cee6f98b08a"
format_stamp: "Formatted at 2026-09-24 20:07:47 on dist-test-slave-2pm7"
server_key: "ed6191c6bc95593e195307062f54e8cc"
server_key_iv: "85c372197f96987ae0a04a064125ada4"
server_key_version: "encryptionkey@0"
I20260924 20:07:47.887049 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "e39d82fd528e4099a7f59cee6f98b08a"
format_stamp: "Formatted at 2026-09-24 20:07:47 on dist-test-slave-2pm7"
server_key: "ed6191c6bc95593e195307062f54e8cc"
server_key_iv: "85c372197f96987ae0a04a064125ada4"
server_key_version: "encryptionkey@0"
I20260924 20:07:47.890983 482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:07:47.893487 497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:47.894711 482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:07:47.894855 482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "e39d82fd528e4099a7f59cee6f98b08a"
format_stamp: "Formatted at 2026-09-24 20:07:47 on dist-test-slave-2pm7"
server_key: "ed6191c6bc95593e195307062f54e8cc"
server_key_iv: "85c372197f96987ae0a04a064125ada4"
server_key_version: "encryptionkey@0"
I20260924 20:07:47.894973 482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:47.917614 482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:47.921097 482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:47.921311 482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:47.928366 482 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:33915
I20260924 20:07:47.928390 549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:33915 every 8 connection(s)
I20260924 20:07:47.929358 482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:07:47.930748 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 482
I20260924 20:07:47.930904 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:07:47.932296 462 external_mini_cluster.cc:1428] Setting key c74bbbec96bf731433792d2c057ec2e6
I20260924 20:07:47.932307 550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:47.938570 550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a: Bootstrap starting.
I20260924 20:07:47.940853 550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:47.941622 550 log.cc:824] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:47.943401 550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a: No bootstrap required, opened a new log
I20260924 20:07:47.946826 550 raft_consensus.cc:348] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39d82fd528e4099a7f59cee6f98b08a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33915 } }
I20260924 20:07:47.947081 550 raft_consensus.cc:374] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:47.947155 550 raft_consensus.cc:729] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39d82fd528e4099a7f59cee6f98b08a, State: Initialized, Role: FOLLOWER
I20260924 20:07:47.947654 550 consensus_queue.cc:260] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39d82fd528e4099a7f59cee6f98b08a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33915 } }
I20260924 20:07:47.947815 550 raft_consensus.cc:388] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:07:47.947899 550 raft_consensus.cc:482] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:07:47.948029 550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:47.949059 550 raft_consensus.cc:504] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39d82fd528e4099a7f59cee6f98b08a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33915 } }
Sep 24 20:07:47 dist-test-slave-2pm7 krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:47.949453 550 leader_election.cc:302] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e39d82fd528e4099a7f59cee6f98b08a; no voters:
I20260924 20:07:47.949847 550 leader_election.cc:288] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:07:47.949955 555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:47.950202 555 raft_consensus.cc:686] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [term 1 LEADER]: Becoming Leader. State: Replica: e39d82fd528e4099a7f59cee6f98b08a, State: Running, Role: LEADER
I20260924 20:07:47.950551 555 consensus_queue.cc:237] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39d82fd528e4099a7f59cee6f98b08a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33915 } }
I20260924 20:07:47.951277 550 sys_catalog.cc:564] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:07:47.952571 557 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e39d82fd528e4099a7f59cee6f98b08a. Latest consensus state: current_term: 1 leader_uuid: "e39d82fd528e4099a7f59cee6f98b08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39d82fd528e4099a7f59cee6f98b08a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33915 } } }
I20260924 20:07:47.952736 557 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:47.952571 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e39d82fd528e4099a7f59cee6f98b08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39d82fd528e4099a7f59cee6f98b08a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33915 } } }
I20260924 20:07:47.952797 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:47.953147 564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:07:47.956604 564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:07:47.956926 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:47.935476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.0.115.190:33915)
0924 20:07:47.935916 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:47.935931 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:47.936961 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 20:07:47.940158 (+ 3197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:47.940176 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:47.940482 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:47.940812 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:47.942973 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:47.944082 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:47.945270 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:47.945770 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:47.945954 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:47.950341 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:47.950347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:47.950361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:47.950393 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:47.952699 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:47.953425 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:47.953435 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:47.953443 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:47.953629 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:47.954138 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:47.954141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:47.954143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:47.954657 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:47.954849 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:47.955653 (+ 804us) server_negotiation.cc:299] Negotiation successful
0924 20:07:47.955933 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":149,"threads_started":1}
I20260924 20:07:47.963675 564 catalog_manager.cc:1347] Generated new cluster ID: d341271ce16e419995afaa2db4f89dc8
I20260924 20:07:47.963752 564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:07:47.975294 564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:07:47.976646 564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:07:47.982904 564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e39d82fd528e4099a7f59cee6f98b08a: Generated new TSK 0
I20260924 20:07:47.983726 564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:07:48.018603 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33915
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:48.132185 576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:48.132488 576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:48.132561 576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:48.136792 576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:48.136879 576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:48.136976 576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:07:48.142904 576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:48.144117 576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:48.145219 576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:48.155193 582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:48.155212 584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:48.155218 581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:48.155516 576 server_base.cc:1034] running on GCE node
I20260924 20:07:48.156090 576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:48.156824 576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:48.157989 576 hybrid_clock.cc:648] HybridClock initialized: now 1790280468157969 us; error 37 us; skew 500 ppm
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:48.160786 576 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:07:48.161716 576 webserver.cc:460] Webserver started at http://127.0.115.129:34113/ using document root <none> and password file <none>
I20260924 20:07:48.162298 576 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:48.162389 576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:48.162627 576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:48.164915 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "afcc7539ebce4e0aa52bd26e19a74ecc"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "c7a011b0877b1d23c4449fb23f92a8bf"
server_key_iv: "b1272069b9067673a61830312853ba17"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.165501 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "afcc7539ebce4e0aa52bd26e19a74ecc"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "c7a011b0877b1d23c4449fb23f92a8bf"
server_key_iv: "b1272069b9067673a61830312853ba17"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.169355 576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:07:48.172012 591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.173275 576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:07:48.173421 576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "afcc7539ebce4e0aa52bd26e19a74ecc"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "c7a011b0877b1d23c4449fb23f92a8bf"
server_key_iv: "b1272069b9067673a61830312853ba17"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.173556 576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:48.201009 576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:48.204375 576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:48.204587 576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:48.205282 576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:07:48.206161 576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:48.206233 576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.206307 576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:48.206350 576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.215358 576 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:35945
I20260924 20:07:48.215406 704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:35945 every 8 connection(s)
I20260924 20:07:48.216346 576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:07:48.225622 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 576
I20260924 20:07:48.225742 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:07:48.226028 462 external_mini_cluster.cc:1428] Setting key ed8a3b9aad513709ee6eb59815b88295
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:48.231796 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.218375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:59215 (local address 127.0.115.190:33915)
0924 20:07:48.218520 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:48.218524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:48.219470 (+ 946us) server_negotiation.cc:407] Connection header received
0924 20:07:48.221026 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:48.221029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:48.221092 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:48.221174 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:48.222438 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.222989 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.224277 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.224547 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.224668 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.227630 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:48.227633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:48.227635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:48.227657 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:48.229299 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.229833 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.229840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.229843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.229939 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.230298 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.230301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.230303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.230574 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:48.230679 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:48.231424 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 20:07:48.231577 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:07:48.232650 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.218814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33915 (local address 127.0.115.129:59215)
0924 20:07:48.219315 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:48.219361 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:48.220784 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:48.221367 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:48.221375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:48.221769 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:48.222274 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:48.222289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.223132 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.223137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:48.224142 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:48.224153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.224725 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.224730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:48.224899 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.225546 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:48.225571 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:48.227457 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:48.229425 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:48.229431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:48.229445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:48.229729 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:48.230041 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:48.230043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:48.230044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:48.230202 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:48.230691 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:48.230696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:48.231158 (+ 462us) client_negotiation.cc:769] Sending connection context
0924 20:07:48.231340 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 20:07:48.231626 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":139,"threads_started":1}
I20260924 20:07:48.234084 705 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33915
I20260924 20:07:48.234354 705 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:48.234875 705 heartbeater.cc:511] Master 127.0.115.190:33915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:07:48.236584 514 ts_manager.cc:194] Registered new tserver with Master: afcc7539ebce4e0aa52bd26e19a74ecc (127.0.115.129:35945)
I20260924 20:07:48.237880 514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:59215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:07:48.255020 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33915
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:48.367157 710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:48.367491 710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:48.367574 710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:48.371661 710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:48.371769 710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:48.371869 710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:07:48.377403 710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:48.378680 710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:48.379812 710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:48.389777 715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:48.389806 716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:48.389889 710 server_base.cc:1034] running on GCE node
W20260924 20:07:48.389792 718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:48.390455 710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:48.391181 710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:48.392372 710 hybrid_clock.cc:648] HybridClock initialized: now 1790280468392355 us; error 35 us; skew 500 ppm
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:48.395344 710 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:07:48.396416 710 webserver.cc:460] Webserver started at http://127.0.115.130:42487/ using document root <none> and password file <none>
I20260924 20:07:48.397024 710 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:48.397118 710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:48.397372 710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:48.399610 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "06431870c94746e6a39bbe8f376d9706"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "741af892196a53f5687295b5c9c2f7f5"
server_key_iv: "349564ed633ec10f424bf422f978d1eb"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.400209 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "06431870c94746e6a39bbe8f376d9706"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "741af892196a53f5687295b5c9c2f7f5"
server_key_iv: "349564ed633ec10f424bf422f978d1eb"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.403893 710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:07:48.406370 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.407796 710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:07:48.408003 710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "06431870c94746e6a39bbe8f376d9706"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "741af892196a53f5687295b5c9c2f7f5"
server_key_iv: "349564ed633ec10f424bf422f978d1eb"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.408124 710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:48.440442 710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:48.443969 710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:48.444192 710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:48.444842 710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:07:48.445752 710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:48.445823 710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.445899 710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:48.445942 710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.455478 710 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:41861
I20260924 20:07:48.455502 838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:41861 every 8 connection(s)
I20260924 20:07:48.456458 710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:07:48.461948 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 710
I20260924 20:07:48.462077 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:07:48.462335 462 external_mini_cluster.cc:1428] Setting key 5e30d2b8334079df4258bf9fe3e8dddf
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:07:48.472005 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.458560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:59297 (local address 127.0.115.190:33915)
0924 20:07:48.458714 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:48.458718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:48.459704 (+ 986us) server_negotiation.cc:407] Connection header received
0924 20:07:48.461037 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:48.461040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:48.461086 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:48.461160 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:48.462752 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.463360 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.465064 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.465288 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.465401 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.468190 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:48.468199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:48.468200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:48.468223 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:48.469865 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.470391 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.470394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.470396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.470487 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.470844 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.470847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.470849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.471092 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:48.471208 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:48.471692 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 20:07:48.471813 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:07:48.472934 841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.459010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33915 (local address 127.0.115.130:59297)
0924 20:07:48.459487 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:48.459533 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:48.460843 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:48.461330 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:48.461336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:48.461851 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:48.462562 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:48.462586 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.463489 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.463496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:48.464922 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:48.464938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.465453 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.465459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:48.465682 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.466299 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:48.466316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:48.468027 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:48.469995 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:48.470001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:48.470017 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:48.470280 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:48.470584 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:48.470587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:48.470589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:48.470737 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:48.471197 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:48.471204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:48.471453 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 20:07:48.471638 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 20:07:48.472009 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260924 20:07:48.474283 839 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33915
I20260924 20:07:48.474541 839 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:07:48.475047 839 heartbeater.cc:511] Master 127.0.115.190:33915 requested a full tablet report, sending...
I20260924 20:07:48.476220 514 ts_manager.cc:194] Registered new tserver with Master: 06431870c94746e6a39bbe8f376d9706 (127.0.115.130:41861)
I20260924 20:07:48.476818 514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:59297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:07:48.490909 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33915
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:48.601734 844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:48.602000 844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:48.602083 844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:48.606148 844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:48.606207 844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:48.606315 844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:07:48.611706 844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:48.612829 844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:48.613888 844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:48.623446 850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:48.623442 852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:48.623672 844 server_base.cc:1034] running on GCE node
W20260924 20:07:48.623442 849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:48.624220 844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:48.624897 844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:48.626080 844 hybrid_clock.cc:648] HybridClock initialized: now 1790280468626063 us; error 29 us; skew 500 ppm
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:48.628925 844 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:07:48.629791 844 webserver.cc:460] Webserver started at http://127.0.115.131:44227/ using document root <none> and password file <none>
I20260924 20:07:48.630366 844 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:48.630457 844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:48.630688 844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:48.633616 844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "63c499990af4413996edb814e0f5adaa"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "8299dcc0697aaba2098a61eba5be59da"
server_key_iv: "8e034ff509258b260c33e1064b72a977"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.634279 844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "63c499990af4413996edb814e0f5adaa"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "8299dcc0697aaba2098a61eba5be59da"
server_key_iv: "8e034ff509258b260c33e1064b72a977"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.638005 844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:07:48.640547 859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.641789 844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:07:48.641909 844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "63c499990af4413996edb814e0f5adaa"
format_stamp: "Formatted at 2026-09-24 20:07:48 on dist-test-slave-2pm7"
server_key: "8299dcc0697aaba2098a61eba5be59da"
server_key_iv: "8e034ff509258b260c33e1064b72a977"
server_key_version: "encryptionkey@0"
I20260924 20:07:48.642014 844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:48.654752 844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:48.657634 844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:48.657828 844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:48.658412 844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:07:48.659231 844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:48.659276 844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.659345 844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:48.659382 844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:48.667996 844 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44601
I20260924 20:07:48.668036 972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44601 every 8 connection(s)
I20260924 20:07:48.668907 844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:07:48.677150 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 844
I20260924 20:07:48.677285 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:07:48.677561 462 external_mini_cluster.cc:1428] Setting key a8b3f6ea4350818823a04bc18f9473f0
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:48.682857 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.670773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:46157 (local address 127.0.115.190:33915)
0924 20:07:48.670921 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:48.670925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:48.671817 (+ 892us) server_negotiation.cc:407] Connection header received
0924 20:07:48.673255 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:48.673259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:48.673318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:48.673418 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:48.674743 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.675272 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.676483 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.676707 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.676815 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.679381 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:48.679385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:48.679387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:48.679415 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:48.680902 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.681395 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.681399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.681401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.681481 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.681810 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.681814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.681816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.681998 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:48.682108 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:48.682588 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 20:07:48.682701 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:07:48.683640 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.671140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33915 (local address 127.0.115.131:46157)
0924 20:07:48.671674 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:48.671716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:48.673033 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:48.673616 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:48.673623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:48.674087 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:48.674568 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:48.674582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.675397 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.675405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:48.676352 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:48.676361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.676857 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.676861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:48.677027 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.677674 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:48.677697 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:48.679221 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:48.681017 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:48.681023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:48.681038 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:48.681290 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:48.681576 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:48.681579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:48.681581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:48.681712 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:48.682105 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:48.682111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:48.682350 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 20:07:48.682546 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 20:07:48.682866 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":109,"threads_started":1}
I20260924 20:07:48.684743 973 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33915
I20260924 20:07:48.684948 973 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:48.685369 973 heartbeater.cc:511] Master 127.0.115.190:33915 requested a full tablet report, sending...
I20260924 20:07:48.686292 514 ts_manager.cc:194] Registered new tserver with Master: 63c499990af4413996edb814e0f5adaa (127.0.115.131:44601)
I20260924 20:07:48.686853 514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:46157
I20260924 20:07:48.691991 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:07:48.717829 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.708573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.0.115.129:35945)
0924 20:07:48.708894 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:48.708908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:48.708932 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:07:48.708983 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:48.708988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:48.709177 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:48.709296 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:48.710020 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.710834 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.711770 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.712193 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.712321 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.714632 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:48.714641 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:48.714654 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:48.714676 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:48.716176 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.716596 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.716602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.716603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.716682 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.716905 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.716907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.716908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.717129 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:48.717287 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:48.717518 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 20:07:48.717677 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20260924 20:07:48.718797 619 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35034
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:07:48.743254 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:48.735425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.0.115.129:35945)
0924 20:07:48.735597 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:48.735602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:48.735622 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:07:48.735770 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:48.735773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:48.735817 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:48.735886 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:48.736590 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.737107 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.737913 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:48.738146 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:48.738220 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:48.740592 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:48.740595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:48.740597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:48.740621 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:48.741843 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.742196 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.742199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.742200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.742271 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:48.742486 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:48.742488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:48.742489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:48.742644 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:48.742737 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:48.742928 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 20:07:48.743029 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 20:07:48.744370 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 576
I20260924 20:07:48.751027 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 710
I20260924 20:07:48.759193 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 844
I20260924 20:07:48.764889 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 482
2026-09-24T20:07:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2223 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[987](info): set up 2 sockets
Sep 24 20:07:48 dist-test-slave-2pm7 krb5kdc[987](info): commencing operation
krb5kdc: starting...
W20260924 20:07:49.836154 462 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 20:07:50.084296 462 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.248s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 20:07:50.282626 462 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.198s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 20:07:50.483904 462 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.201s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/test-user.keytab.
W20260924 20:07:50.957931 462 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.474s user 0.000s sys 0.002s
Sep 24 20:07:50 dist-test-slave-2pm7 krb5kdc[987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:07:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
W20260924 20:07:51.818485 462 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.115.190: real 0.837s user 0.002s sys 0.001s
I20260924 20:07:51.819052 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:37161
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:37161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:51.939280 1000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:51.939546 1000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:51.939594 1000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:51.943538 1000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:07:51.943595 1000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:51.943615 1000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:51.943632 1000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:07:51.943647 1000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:07:51.948905 1000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:37161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:51.949908 1000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:51.950934 1000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:51.959947 1005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:51.960140 1000 server_base.cc:1034] running on GCE node
W20260924 20:07:51.959991 1008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:51.959991 1006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:51.960822 1000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:51.962077 1000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:51.963409 1000 hybrid_clock.cc:648] HybridClock initialized: now 1790280471963388 us; error 29 us; skew 500 ppm
Sep 24 20:07:51 dist-test-slave-2pm7 krb5kdc[987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:51.969755 1000 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:07:51.970777 1000 webserver.cc:460] Webserver started at http://127.0.115.190:35213/ using document root <none> and password file <none>
I20260924 20:07:51.971375 1000 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:51.971436 1000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:51.971680 1000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:51.973875 1000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "e74f74f72fc842a985c45e4fcea0ec7b"
format_stamp: "Formatted at 2026-09-24 20:07:51 on dist-test-slave-2pm7"
server_key: "07cdc6480f8bb6b6dbe192c1449dee68"
server_key_iv: "16121ea64b15798e3d680c1789f91504"
server_key_version: "encryptionkey@0"
I20260924 20:07:51.974453 1000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "e74f74f72fc842a985c45e4fcea0ec7b"
format_stamp: "Formatted at 2026-09-24 20:07:51 on dist-test-slave-2pm7"
server_key: "07cdc6480f8bb6b6dbe192c1449dee68"
server_key_iv: "16121ea64b15798e3d680c1789f91504"
server_key_version: "encryptionkey@0"
I20260924 20:07:51.978279 1000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:07:51.980723 1015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:51.981969 1000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:07:51.982108 1000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "e74f74f72fc842a985c45e4fcea0ec7b"
format_stamp: "Formatted at 2026-09-24 20:07:51 on dist-test-slave-2pm7"
server_key: "07cdc6480f8bb6b6dbe192c1449dee68"
server_key_iv: "16121ea64b15798e3d680c1789f91504"
server_key_version: "encryptionkey@0"
I20260924 20:07:51.982237 1000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:51.999186 1000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:52.002225 1000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:52.002436 1000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:52.009896 1000 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:37161
I20260924 20:07:52.009927 1067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:37161 every 8 connection(s)
I20260924 20:07:52.010885 1000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:07:52.013826 1068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:52.016170 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1000
I20260924 20:07:52.016296 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:07:52.016567 462 external_mini_cluster.cc:1428] Setting key 2de7ec6225a19c9cf1cbb8eb6eb7c442
I20260924 20:07:52.020244 1068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b: Bootstrap starting.
I20260924 20:07:52.023267 1068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:52.024271 1068 log.cc:824] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:07:52 dist-test-slave-2pm7 krb5kdc[987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:52.026598 1068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b: No bootstrap required, opened a new log
I20260924 20:07:52.029636 1068 raft_consensus.cc:348] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37161 } }
I20260924 20:07:52.029917 1068 raft_consensus.cc:374] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:52.029989 1068 raft_consensus.cc:729] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74f74f72fc842a985c45e4fcea0ec7b, State: Initialized, Role: FOLLOWER
I20260924 20:07:52.030521 1068 consensus_queue.cc:260] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37161 } }
I20260924 20:07:52.030709 1068 raft_consensus.cc:388] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:07:52.030777 1068 raft_consensus.cc:482] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:07:52.030874 1068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:52.031435 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:52.018047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.0.115.190:37161)
0924 20:07:52.018488 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:52.018503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:52.018538 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:07:52.019302 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:52.019322 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:52.019675 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:52.020042 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:52.020872 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:52.021853 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:52.022729 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:52.023208 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:52.023445 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:52.026293 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:52.026298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:52.026311 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:52.026337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:52.028568 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:52.029012 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:52.029018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:52.029022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:52.029135 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:52.029418 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:52.029422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:52.029423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:52.029819 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:52.030055 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:52.030292 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 20:07:52.030582 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260924 20:07:52.104938 1068 raft_consensus.cc:504] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37161 } }
I20260924 20:07:52.105516 1068 leader_election.cc:302] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e74f74f72fc842a985c45e4fcea0ec7b; no voters:
I20260924 20:07:52.105944 1068 leader_election.cc:288] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:07:52.106108 1073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:52.106386 1073 raft_consensus.cc:686] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [term 1 LEADER]: Becoming Leader. State: Replica: e74f74f72fc842a985c45e4fcea0ec7b, State: Running, Role: LEADER
I20260924 20:07:52.106846 1073 consensus_queue.cc:237] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37161 } }
I20260924 20:07:52.107400 1068 sys_catalog.cc:564] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:07:52.108904 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37161 } } }
I20260924 20:07:52.109037 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:52.109519 1075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e74f74f72fc842a985c45e4fcea0ec7b. Latest consensus state: current_term: 1 leader_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74f74f72fc842a985c45e4fcea0ec7b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37161 } } }
I20260924 20:07:52.109709 1075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:52.109989 1083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:07:52.113214 1083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:07:52.118368 1083 catalog_manager.cc:1347] Generated new cluster ID: a74f2987c3dd48f48aa390026f3c23ec
I20260924 20:07:52.118443 1083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:07:52.125449 1083 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:07:52.126329 1083 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:07:52.131601 1083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e74f74f72fc842a985c45e4fcea0ec7b: Generated new TSK 0
I20260924 20:07:52.132433 1083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
W20260924 20:07:53.039100 462 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.115.129: real 0.891s user 0.000s sys 0.003s
I20260924 20:07:53.039594 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:37161
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:53.160198 1094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:53.160468 1094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:53.160548 1094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:53.164806 1094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:53.164891 1094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:53.165006 1094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:07:53.170895 1094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:37161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:53.172091 1094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:53.173188 1094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:53.183254 1100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:53.183363 1102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:53.183387 1099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:53.183506 1094 server_base.cc:1034] running on GCE node
I20260924 20:07:53.184039 1094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:53.184772 1094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:53.185966 1094 hybrid_clock.cc:648] HybridClock initialized: now 1790280473185945 us; error 31 us; skew 500 ppm
Sep 24 20:07:53 dist-test-slave-2pm7 krb5kdc[987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:53.188998 1094 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:07:53.189873 1094 webserver.cc:460] Webserver started at http://127.0.115.129:42121/ using document root <none> and password file <none>
I20260924 20:07:53.190443 1094 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:53.190505 1094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:53.190750 1094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:53.192879 1094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "76fdbe7445f244599f194eb90dc77348"
format_stamp: "Formatted at 2026-09-24 20:07:53 on dist-test-slave-2pm7"
server_key: "3168187a30eab5f88e0dd256938fc4ec"
server_key_iv: "b4468415bbe1b8820d05fed212f9e89d"
server_key_version: "encryptionkey@0"
I20260924 20:07:53.193455 1094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "76fdbe7445f244599f194eb90dc77348"
format_stamp: "Formatted at 2026-09-24 20:07:53 on dist-test-slave-2pm7"
server_key: "3168187a30eab5f88e0dd256938fc4ec"
server_key_iv: "b4468415bbe1b8820d05fed212f9e89d"
server_key_version: "encryptionkey@0"
I20260924 20:07:53.197295 1094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:07:53.199872 1109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:53.201257 1094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:07:53.201393 1094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "76fdbe7445f244599f194eb90dc77348"
format_stamp: "Formatted at 2026-09-24 20:07:53 on dist-test-slave-2pm7"
server_key: "3168187a30eab5f88e0dd256938fc4ec"
server_key_iv: "b4468415bbe1b8820d05fed212f9e89d"
server_key_version: "encryptionkey@0"
I20260924 20:07:53.201507 1094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:53.214035 1094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:53.217870 1094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:53.218083 1094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:53.218879 1094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:07:53.220132 1094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:53.220202 1094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:53.220268 1094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:53.220304 1094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:53.229386 1094 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:45809
I20260924 20:07:53.229454 1222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:45809 every 8 connection(s)
I20260924 20:07:53.230304 1094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:07:53.236486 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1094
I20260924 20:07:53.236601 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:07:53.236838 462 external_mini_cluster.cc:1428] Setting key 1b4232501ac09fd2a427f87cb9a5eec6
Sep 24 20:07:53 dist-test-slave-2pm7 krb5kdc[987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:53.245178 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:53.232238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:54509 (local address 127.0.115.190:37161)
0924 20:07:53.232505 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:53.232509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:53.233254 (+ 745us) server_negotiation.cc:407] Connection header received
0924 20:07:53.234656 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:53.234660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:53.234725 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:53.234831 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:53.236076 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:53.236724 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:53.238223 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:53.238540 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:53.238653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:53.241162 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:53.241165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:53.241167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:53.241189 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:53.242698 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:53.243367 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:53.243370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:53.243371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:53.243446 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:53.243844 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:53.243846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:53.243848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:53.244062 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:53.244145 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:53.244881 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 20:07:53.245025 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260924 20:07:53.246359 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:53.232631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:37161 (local address 127.0.115.129:54509)
0924 20:07:53.233097 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:53.233140 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:53.234435 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:53.235012 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:53.235019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:53.235387 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:53.235903 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:53.235917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:53.236844 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:53.236848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:53.238087 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:53.238118 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:53.238680 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:53.238683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:53.238842 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:53.239445 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:53.239461 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:53.240994 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:53.242822 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:53.242832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:53.242851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:53.243250 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:53.243538 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:53.243544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:53.243548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:53.243749 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:53.244174 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:53.244183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:53.244584 (+ 401us) client_negotiation.cc:769] Sending connection context
0924 20:07:53.244811 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 20:07:53.245187 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:07:53.247789 1223 heartbeater.cc:347] Connected to a master server at 127.0.115.190:37161
I20260924 20:07:53.248054 1223 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:53.248545 1223 heartbeater.cc:511] Master 127.0.115.190:37161 requested a full tablet report, sending...
I20260924 20:07:53.250120 1032 ts_manager.cc:194] Registered new tserver with Master: 76fdbe7445f244599f194eb90dc77348 (127.0.115.129:45809)
I20260924 20:07:53.251497 1032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:54509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
W20260924 20:07:54.247918 462 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.115.130: real 1.009s user 0.000s sys 0.003s
I20260924 20:07:54.248463 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:37161
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:07:54.253808 1223 heartbeater.cc:503] Master 127.0.115.190:37161 was elected leader, sending a full tablet report...
W20260924 20:07:54.361282 1229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:54.361554 1229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:54.361637 1229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:54.365851 1229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:54.365914 1229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:54.366019 1229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:07:54.371490 1229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:37161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:54.372695 1229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:54.373895 1229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:54.383991 1235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:54.384042 1237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:54.383982 1234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:54.384228 1229 server_base.cc:1034] running on GCE node
I20260924 20:07:54.384817 1229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:54.385548 1229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:54.386741 1229 hybrid_clock.cc:648] HybridClock initialized: now 1790280474386728 us; error 36 us; skew 500 ppm
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:54.389599 1229 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:07:54.390563 1229 webserver.cc:460] Webserver started at http://127.0.115.130:45177/ using document root <none> and password file <none>
I20260924 20:07:54.391105 1229 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:54.391162 1229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:54.391330 1229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:54.393514 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "a20806d5f6d24cb4a44206bcbb327ba1"
format_stamp: "Formatted at 2026-09-24 20:07:54 on dist-test-slave-2pm7"
server_key: "46eab101c76d1f4054c0f6ca12851409"
server_key_iv: "3c47160e77681e023fd3f3cbc18b004d"
server_key_version: "encryptionkey@0"
I20260924 20:07:54.394090 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "a20806d5f6d24cb4a44206bcbb327ba1"
format_stamp: "Formatted at 2026-09-24 20:07:54 on dist-test-slave-2pm7"
server_key: "46eab101c76d1f4054c0f6ca12851409"
server_key_iv: "3c47160e77681e023fd3f3cbc18b004d"
server_key_version: "encryptionkey@0"
I20260924 20:07:54.397915 1229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:07:54.400420 1244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:54.401670 1229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:07:54.401806 1229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "a20806d5f6d24cb4a44206bcbb327ba1"
format_stamp: "Formatted at 2026-09-24 20:07:54 on dist-test-slave-2pm7"
server_key: "46eab101c76d1f4054c0f6ca12851409"
server_key_iv: "3c47160e77681e023fd3f3cbc18b004d"
server_key_version: "encryptionkey@0"
I20260924 20:07:54.401908 1229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:54.420099 1229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:54.423175 1229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:54.423386 1229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:54.424100 1229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:07:54.424981 1229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:54.425029 1229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:54.425081 1229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:54.425097 1229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:54.434291 1229 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:44637
I20260924 20:07:54.434316 1357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:44637 every 8 connection(s)
I20260924 20:07:54.435216 1229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:54.445176 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1229
I20260924 20:07:54.445295 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:07:54.445624 462 external_mini_cluster.cc:1428] Setting key 6cc09b2bed47356a7eeadce038af3e23
I20260924 20:07:54.449441 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.437080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:51623 (local address 127.0.115.190:37161)
0924 20:07:54.437301 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.437305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.438036 (+ 731us) server_negotiation.cc:407] Connection header received
0924 20:07:54.439396 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.439399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.439461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.439572 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:54.440861 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.441415 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.442563 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.442799 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.442909 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.445563 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:54.445569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:54.445574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:54.445606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:54.447296 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.447805 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.447809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.447811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.447897 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.448280 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.448284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.448286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.448471 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:54.448574 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.449123 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.449258 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260924 20:07:54.450259 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.437417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:37161 (local address 127.0.115.130:51623)
0924 20:07:54.437888 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:54.437931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:54.439187 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:54.439759 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:54.439766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:54.440221 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:54.440710 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.440723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.441541 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.441544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.442435 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.442444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.442949 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.442954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.443119 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.443714 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:54.443731 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:54.445257 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:54.447411 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:54.447417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:54.447433 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:54.447702 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:54.448035 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:54.448038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:54.448041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:54.448184 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:54.448573 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:54.448582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:54.448863 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 20:07:54.449070 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 20:07:54.449403 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260924 20:07:54.451483 1358 heartbeater.cc:347] Connected to a master server at 127.0.115.190:37161
I20260924 20:07:54.451750 1358 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:54.452296 1358 heartbeater.cc:511] Master 127.0.115.190:37161 requested a full tablet report, sending...
I20260924 20:07:54.453457 1032 ts_manager.cc:194] Registered new tserver with Master: a20806d5f6d24cb4a44206bcbb327ba1 (127.0.115.130:44637)
I20260924 20:07:54.454049 1032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:51623
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:07:54.474852 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:37161
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:54.588719 1364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:54.589020 1364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:54.589089 1364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:54.593292 1364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:54.593375 1364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:54.593463 1364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:07:54.598744 1364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:37161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:54.599838 1364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:54.600966 1364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:54.610782 1369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:54.610771 1370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:54.610771 1372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:54.611043 1364 server_base.cc:1034] running on GCE node
I20260924 20:07:54.611475 1364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:54.612217 1364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:54.613402 1364 hybrid_clock.cc:648] HybridClock initialized: now 1790280474613367 us; error 49 us; skew 500 ppm
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:54.616192 1364 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:07:54.617023 1364 webserver.cc:460] Webserver started at http://127.0.115.131:39273/ using document root <none> and password file <none>
I20260924 20:07:54.617555 1364 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:54.617621 1364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:54.617784 1364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:54.619765 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "f7a351e041bf4f39ad6fa8fbb77ec574"
format_stamp: "Formatted at 2026-09-24 20:07:54 on dist-test-slave-2pm7"
server_key: "539a7c898039ffbd3532295b95df6ec2"
server_key_iv: "d3492a325445670204d843a00ccb7f23"
server_key_version: "encryptionkey@0"
I20260924 20:07:54.620337 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "f7a351e041bf4f39ad6fa8fbb77ec574"
format_stamp: "Formatted at 2026-09-24 20:07:54 on dist-test-slave-2pm7"
server_key: "539a7c898039ffbd3532295b95df6ec2"
server_key_iv: "d3492a325445670204d843a00ccb7f23"
server_key_version: "encryptionkey@0"
I20260924 20:07:54.624151 1364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:07:54.626502 1379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:54.627665 1364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:07:54.627758 1364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "f7a351e041bf4f39ad6fa8fbb77ec574"
format_stamp: "Formatted at 2026-09-24 20:07:54 on dist-test-slave-2pm7"
server_key: "539a7c898039ffbd3532295b95df6ec2"
server_key_iv: "d3492a325445670204d843a00ccb7f23"
server_key_version: "encryptionkey@0"
I20260924 20:07:54.627840 1364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:54.648567 1364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:54.651362 1364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:54.651535 1364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:54.652141 1364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:07:54.652940 1364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:54.652983 1364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:54.653026 1364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:54.653048 1364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:54.662701 1364 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44781
I20260924 20:07:54.662725 1492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44781 every 8 connection(s)
I20260924 20:07:54.663578 1364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:07:54.671518 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1364
I20260924 20:07:54.671713 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:07:54.672119 462 external_mini_cluster.cc:1428] Setting key 79b056a3aa13d5971f180371bff544e8
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:54.677464 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.665374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:37469 (local address 127.0.115.190:37161)
0924 20:07:54.665518 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.665523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.666283 (+ 760us) server_negotiation.cc:407] Connection header received
0924 20:07:54.667833 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.667837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.667897 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.668022 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:54.669232 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.669775 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.670916 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.671154 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.671264 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.674094 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:54.674098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:54.674101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:54.674127 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:54.675478 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.676000 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.676006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.676008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.676091 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.676421 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.676426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.676428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.676606 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:54.676708 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.677170 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.677303 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:07:54.678208 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.665697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:37161 (local address 127.0.115.131:37469)
0924 20:07:54.666136 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:54.666179 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:54.667618 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:54.668246 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:54.668254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:54.668611 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:54.669083 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.669097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.669899 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.669903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.670780 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.670790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.671325 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.671329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.671486 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.672171 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:54.672197 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:54.673931 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:54.675608 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:54.675614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:54.675628 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:54.675883 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:54.676187 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:54.676190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:54.676192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:54.676328 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:54.676713 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:54.676722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:54.676951 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 20:07:54.677129 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 20:07:54.677431 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260924 20:07:54.679412 1493 heartbeater.cc:347] Connected to a master server at 127.0.115.190:37161
I20260924 20:07:54.679629 1493 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:54.680166 1493 heartbeater.cc:511] Master 127.0.115.190:37161 requested a full tablet report, sending...
I20260924 20:07:54.681193 1032 ts_manager.cc:194] Registered new tserver with Master: f7a351e041bf4f39ad6fa8fbb77ec574 (127.0.115.131:44781)
I20260924 20:07:54.681769 1032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:37469
I20260924 20:07:54.685869 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:07:54.696502 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.689401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.0.115.190:37161)
0924 20:07:54.689542 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.689547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.689702 (+ 155us) server_negotiation.cc:407] Connection header received
0924 20:07:54.689857 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.689861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.689902 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.689981 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:54.690667 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.691167 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.691981 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.692195 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.692306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.693565 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:54.693569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:54.693574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:54.693602 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:54.694846 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.695256 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.695264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.695265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.695346 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.695614 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.695618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.695619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.695827 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:54.695982 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.696227 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.696342 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:07:54.701828 1032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:07:54.705066 1032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:07:54.720059 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.714173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:44637 (local address 127.0.0.1:40706)
0924 20:07:54.714807 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:54.714859 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:54.714982 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:54.715843 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:54.715849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:54.715871 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:54.716052 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.716058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.717255 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.717262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.718319 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.718330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.719427 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.719431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.719565 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.719613 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 20:07:54.719736 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 20:07:54.719861 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":116,"threads_started":1}
I20260924 20:07:54.720081 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.714314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.0.115.130:44637)
0924 20:07:54.715291 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.715307 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.715330 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:07:54.715414 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.715423 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.715641 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.715839 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:54.716161 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.717129 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.718456 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.719298 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.719405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.719544 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.719791 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.719895 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":110,"threads_started":1}
I20260924 20:07:54.721004 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.714914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:45809 (local address 127.0.0.1:37810)
0924 20:07:54.715732 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:54.715746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:54.715851 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:54.716437 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:54.716440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:54.716459 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:54.716547 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.716552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.718063 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.718067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.719115 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.719126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.720570 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.720578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.720663 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.720683 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:07:54.720785 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 20:07:54.720864 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"spinlock_wait_cycles":8192,"thread_start_us":85,"threads_started":1}
I20260924 20:07:54.721446 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.715129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.0.115.129:45809)
0924 20:07:54.715588 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.715595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.715990 (+ 395us) server_negotiation.cc:407] Connection header received
0924 20:07:54.716078 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.716083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.716289 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.716447 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:54.716667 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.717918 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.719246 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.720220 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.720351 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.720493 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.721203 (+ 710us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.721305 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":130,"threads_started":1}
I20260924 20:07:54.722011 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.714345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44781 (local address 127.0.0.1:41882)
0924 20:07:54.715718 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:54.715733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:54.715851 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:54.716660 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:54.716663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:54.716676 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:54.716744 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.716748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.718677 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.718680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.719731 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.719745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.721570 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.721574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.721675 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.721709 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 20:07:54.721797 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:07:54.721875 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"spinlock_wait_cycles":5120,"thread_start_us":85,"threads_started":1}
I20260924 20:07:54.722541 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.715310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.0.115.131:44781)
0924 20:07:54.715714 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.715728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.715991 (+ 263us) server_negotiation.cc:407] Connection header received
0924 20:07:54.716083 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.716097 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.716284 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.716447 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:54.716857 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.718538 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.719873 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.720649 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.720755 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.720863 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.722260 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.722370 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":111,"threads_started":1}
I20260924 20:07:54.723845 1292 tablet_service.cc:1467] Processing CreateTablet for tablet ab51ad59e96744ae9e931372b7e0cfd3 (DEFAULT_TABLE table=test-table [id=05b980b2a4184330993c9b569923c0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:54.724478 1157 tablet_service.cc:1467] Processing CreateTablet for tablet ab51ad59e96744ae9e931372b7e0cfd3 (DEFAULT_TABLE table=test-table [id=05b980b2a4184330993c9b569923c0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:54.725060 1292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab51ad59e96744ae9e931372b7e0cfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:54.724936 1427 tablet_service.cc:1467] Processing CreateTablet for tablet ab51ad59e96744ae9e931372b7e0cfd3 (DEFAULT_TABLE table=test-table [id=05b980b2a4184330993c9b569923c0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:54.725482 1157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab51ad59e96744ae9e931372b7e0cfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:54.725906 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab51ad59e96744ae9e931372b7e0cfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:54.732097 1511 tablet_bootstrap.cc:492] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574: Bootstrap starting.
I20260924 20:07:54.733669 1512 tablet_bootstrap.cc:492] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348: Bootstrap starting.
I20260924 20:07:54.733861 1513 tablet_bootstrap.cc:492] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1: Bootstrap starting.
I20260924 20:07:54.734834 1511 tablet_bootstrap.cc:654] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:54.735407 1512 tablet_bootstrap.cc:654] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:54.736157 1512 log.cc:824] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:54.736294 1511 log.cc:824] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:54.736778 1513 tablet_bootstrap.cc:654] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:54.737548 1513 log.cc:824] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:54.738274 1512 tablet_bootstrap.cc:492] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348: No bootstrap required, opened a new log
I20260924 20:07:54.738474 1512 ts_tablet_manager.cc:1397] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:07:54.739303 1511 tablet_bootstrap.cc:492] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574: No bootstrap required, opened a new log
I20260924 20:07:54.739506 1511 ts_tablet_manager.cc:1397] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 20:07:54.739565 1513 tablet_bootstrap.cc:492] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1: No bootstrap required, opened a new log
I20260924 20:07:54.739810 1513 ts_tablet_manager.cc:1397] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:07:54.741894 1512 raft_consensus.cc:348] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.742126 1512 raft_consensus.cc:374] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:54.742184 1512 raft_consensus.cc:729] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fdbe7445f244599f194eb90dc77348, State: Initialized, Role: FOLLOWER
I20260924 20:07:54.742642 1512 consensus_queue.cc:260] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.743660 1512 ts_tablet_manager.cc:1428] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:07:54.743904 1511 raft_consensus.cc:348] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.743904 1513 raft_consensus.cc:348] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.744211 1511 raft_consensus.cc:374] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:54.744211 1513 raft_consensus.cc:374] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:54.744279 1513 raft_consensus.cc:729] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20806d5f6d24cb4a44206bcbb327ba1, State: Initialized, Role: FOLLOWER
I20260924 20:07:54.744287 1511 raft_consensus.cc:729] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a351e041bf4f39ad6fa8fbb77ec574, State: Initialized, Role: FOLLOWER
I20260924 20:07:54.744768 1513 consensus_queue.cc:260] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.744769 1511 consensus_queue.cc:260] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.745682 1358 heartbeater.cc:503] Master 127.0.115.190:37161 was elected leader, sending a full tablet report...
I20260924 20:07:54.745814 1513 ts_tablet_manager.cc:1428] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:07:54.745814 1511 ts_tablet_manager.cc:1428] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:07:54.746220 1493 heartbeater.cc:503] Master 127.0.115.190:37161 was elected leader, sending a full tablet report...
I20260924 20:07:54.789069 1517 raft_consensus.cc:482] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:07:54.789273 1517 raft_consensus.cc:504] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.790410 1517 leader_election.cc:288] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a20806d5f6d24cb4a44206bcbb327ba1 (127.0.115.130:44637), f7a351e041bf4f39ad6fa8fbb77ec574 (127.0.115.131:44781)
I20260924 20:07:54.794595 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.790824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:35805 (local address 127.0.115.131:44781)
0924 20:07:54.790971 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.790976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.791331 (+ 355us) server_negotiation.cc:407] Connection header received
0924 20:07:54.791485 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.791488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.791536 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.791612 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:54.791909 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.792464 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.793609 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.794011 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.794138 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.794191 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.794381 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.794449 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:07:54.794618 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.791106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:55443 (local address 127.0.115.130:44637)
0924 20:07:54.791276 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.791280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.791295 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:07:54.791401 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.791404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.791451 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.791534 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:54.791805 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.792414 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.793603 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.794006 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.794143 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.794199 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.794411 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.794478 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:07:54.794634 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.790697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44781 (local address 127.0.115.129:35805)
0924 20:07:54.791243 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:54.791255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:54.791372 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:54.791634 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:54.791637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:54.791654 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:54.791777 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.791782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.792583 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.792587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.793479 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.793489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.794131 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.794135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.794231 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.794275 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 20:07:54.794365 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:07:54.794440 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":94,"thread_start_us":117,"threads_started":1}
I20260924 20:07:54.794641 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.790697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:44637 (local address 127.0.115.129:55443)
0924 20:07:54.791176 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:54.791192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:54.791307 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:54.791546 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:54.791549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:54.791565 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:54.791693 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.791704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.792518 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.792523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.793462 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:54.793472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.794137 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.794141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:54.794246 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.794299 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 20:07:54.794395 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 20:07:54.794467 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":82,"thread_start_us":124,"threads_started":1}
I20260924 20:07:54.795370 1312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51ad59e96744ae9e931372b7e0cfd3" candidate_uuid: "76fdbe7445f244599f194eb90dc77348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" is_pre_election: true
I20260924 20:07:54.795492 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51ad59e96744ae9e931372b7e0cfd3" candidate_uuid: "76fdbe7445f244599f194eb90dc77348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" is_pre_election: true
I20260924 20:07:54.795755 1447 raft_consensus.cc:2454] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fdbe7445f244599f194eb90dc77348 in term 0.
I20260924 20:07:54.795758 1312 raft_consensus.cc:2454] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fdbe7445f244599f194eb90dc77348 in term 0.
I20260924 20:07:54.796317 1113 leader_election.cc:302] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 76fdbe7445f244599f194eb90dc77348, a20806d5f6d24cb4a44206bcbb327ba1, f7a351e041bf4f39ad6fa8fbb77ec574; no voters:
I20260924 20:07:54.796582 1517 raft_consensus.cc:2781] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:07:54.796669 1517 raft_consensus.cc:482] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:07:54.796720 1517 raft_consensus.cc:3037] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:54.797791 1517 raft_consensus.cc:504] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.798153 1517 leader_election.cc:288] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [CANDIDATE]: Term 1 election: Requested vote from peers a20806d5f6d24cb4a44206bcbb327ba1 (127.0.115.130:44637), f7a351e041bf4f39ad6fa8fbb77ec574 (127.0.115.131:44781)
I20260924 20:07:54.798528 1312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51ad59e96744ae9e931372b7e0cfd3" candidate_uuid: "76fdbe7445f244599f194eb90dc77348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20806d5f6d24cb4a44206bcbb327ba1"
I20260924 20:07:54.798530 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51ad59e96744ae9e931372b7e0cfd3" candidate_uuid: "76fdbe7445f244599f194eb90dc77348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574"
I20260924 20:07:54.798672 1447 raft_consensus.cc:3037] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:54.798643 1312 raft_consensus.cc:3037] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:54.799746 1447 raft_consensus.cc:2454] T ab51ad59e96744ae9e931372b7e0cfd3 P f7a351e041bf4f39ad6fa8fbb77ec574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fdbe7445f244599f194eb90dc77348 in term 1.
I20260924 20:07:54.799782 1312 raft_consensus.cc:2454] T ab51ad59e96744ae9e931372b7e0cfd3 P a20806d5f6d24cb4a44206bcbb327ba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fdbe7445f244599f194eb90dc77348 in term 1.
I20260924 20:07:54.800093 1113 leader_election.cc:302] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76fdbe7445f244599f194eb90dc77348, f7a351e041bf4f39ad6fa8fbb77ec574; no voters:
I20260924 20:07:54.800275 1517 raft_consensus.cc:2781] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:54.800539 1517 raft_consensus.cc:686] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [term 1 LEADER]: Becoming Leader. State: Replica: 76fdbe7445f244599f194eb90dc77348, State: Running, Role: LEADER
I20260924 20:07:54.800918 1517 consensus_queue.cc:237] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } }
I20260924 20:07:54.804330 1030 catalog_manager.cc:5495] T ab51ad59e96744ae9e931372b7e0cfd3 P 76fdbe7445f244599f194eb90dc77348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76fdbe7445f244599f194eb90dc77348 (127.0.115.129). New cstate: current_term: 1 leader_uuid: "76fdbe7445f244599f194eb90dc77348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20806d5f6d24cb4a44206bcbb327ba1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7a351e041bf4f39ad6fa8fbb77ec574" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fdbe7445f244599f194eb90dc77348" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45809 } health_report { overall_health: HEALTHY } } }
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:07:54.834232 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.825605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.0.115.129:45809)
0924 20:07:54.825784 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.825788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.825856 (+ 68us) server_negotiation.cc:407] Connection header received
0924 20:07:54.825969 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.825971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.826013 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.826081 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:54.826711 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.827225 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.828079 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.828298 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.828378 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.830791 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:54.830804 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:54.830820 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:54.830844 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:54.832639 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.833039 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.833045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.833046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.833130 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.833357 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.833359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.833361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.833610 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:54.833773 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.833899 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.833993 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:07:54.863426 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.851793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.0.115.129:45809)
0924 20:07:54.852010 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.852014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.852093 (+ 79us) server_negotiation.cc:407] Connection header received
0924 20:07:54.852222 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.852225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.852275 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.852364 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:54.853067 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.854042 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.855863 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.856354 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.856457 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.860240 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:54.860243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:54.860245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:54.860268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:54.861739 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.862171 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.862176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.862178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.862258 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.862533 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.862537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.862538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.862708 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:54.862842 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.863137 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.863264 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:07:54.892721 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:54.882534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.0.115.129:45809)
0924 20:07:54.882706 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:54.882711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:54.882789 (+ 78us) server_negotiation.cc:407] Connection header received
0924 20:07:54.882917 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:54.882923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:54.882988 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:54.883094 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:54.883741 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.884763 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.885686 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:54.886128 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:54.886286 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:54.888957 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:54.888963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:54.888969 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:54.889006 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:54.890787 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.891232 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.891237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.891241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.891341 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:54.891664 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:54.891669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:54.891673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:54.891914 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:54.892080 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:54.892314 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 20:07:54.892468 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 20:07:54.894641 1137 tablet_service.cc:2908] Rejecting scan request for tablet ab51ad59e96744ae9e931372b7e0cfd3: Uninitialized: clean time has not yet been initialized
I20260924 20:07:54.896838 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1094
I20260924 20:07:54.903906 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1229
I20260924 20:07:54.909772 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1364
I20260924 20:07:54.915349 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1000
2026-09-24T20:07:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6150 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[1536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[1536](info): set up 2 sockets
Sep 24 20:07:54 dist-test-slave-2pm7 krb5kdc[1536](info): commencing operation
krb5kdc: starting...
W20260924 20:07:55.947863 462 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:07:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:07:56.071853 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46337
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:56.184417 1549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:56.184727 1549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:56.184806 1549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:56.188836 1549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:07:56.188913 1549 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:07:56.188941 1549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:56.188982 1549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:56.189020 1549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:07:56.189050 1549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:07:56.194434 1549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:46337
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:56.195544 1549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:56.196652 1549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:56.205471 1557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:56.205580 1554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:56.205720 1549 server_base.cc:1034] running on GCE node
W20260924 20:07:56.205539 1555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:56.206455 1549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:56.207718 1549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:56.208909 1549 hybrid_clock.cc:648] HybridClock initialized: now 1790280476208895 us; error 55 us; skew 500 ppm
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:56.211644 1549 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:07:56.212587 1549 webserver.cc:460] Webserver started at http://127.0.115.190:34229/ using document root <none> and password file <none>
I20260924 20:07:56.213146 1549 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:56.213231 1549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:56.213454 1549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:56.215531 1549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "b9b72457957e4349bf79eb081bc85c03"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "c71aad1dd030b649812dedf136cf7b97"
server_key_iv: "9e5c982416b4e6db4ac1b7e53b962da3"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.216151 1549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "b9b72457957e4349bf79eb081bc85c03"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "c71aad1dd030b649812dedf136cf7b97"
server_key_iv: "9e5c982416b4e6db4ac1b7e53b962da3"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.220233 1549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:07:56.222857 1564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:56.224180 1549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:07:56.224313 1549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "b9b72457957e4349bf79eb081bc85c03"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "c71aad1dd030b649812dedf136cf7b97"
server_key_iv: "9e5c982416b4e6db4ac1b7e53b962da3"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.224424 1549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:56.252825 1549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:56.255825 1549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:56.256067 1549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:56.264038 1549 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:46337
I20260924 20:07:56.264058 1626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:46337 every 8 connection(s)
I20260924 20:07:56.265043 1549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:07:56.267884 1627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:56.268429 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1549
I20260924 20:07:56.268548 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:07:56.268831 462 external_mini_cluster.cc:1428] Setting key ed308737fa1a9c63ab07c7db1ce551bd
I20260924 20:07:56.274864 1627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03: Bootstrap starting.
I20260924 20:07:56.277439 1627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03: Neither blocks nor log segments found. Creating new log.
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:56.278389 1627 log.cc:824] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:56.280679 1627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03: No bootstrap required, opened a new log
I20260924 20:07:56.283790 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.270308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.0.115.190:46337)
0924 20:07:56.270857 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:56.270873 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:56.270911 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:07:56.271595 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:56.271612 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:56.272006 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:56.272383 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:56.273176 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.274190 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.275116 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.275598 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.275763 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.278573 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:56.278579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:56.278592 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:56.278620 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:56.280978 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.281475 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.281481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.281486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.281594 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.281901 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.281903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.281904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.282296 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:56.282441 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:56.282734 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 20:07:56.283038 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":122,"threads_started":1}
I20260924 20:07:56.284307 1627 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b72457957e4349bf79eb081bc85c03" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46337 } }
I20260924 20:07:56.284540 1627 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:56.284643 1627 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b72457957e4349bf79eb081bc85c03, State: Initialized, Role: FOLLOWER
I20260924 20:07:56.285173 1627 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b72457957e4349bf79eb081bc85c03" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46337 } }
I20260924 20:07:56.285332 1627 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:07:56.285399 1627 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:07:56.285497 1627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:56.286520 1627 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b72457957e4349bf79eb081bc85c03" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46337 } }
I20260924 20:07:56.286813 1627 leader_election.cc:302] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9b72457957e4349bf79eb081bc85c03; no voters:
I20260924 20:07:56.287077 1627 leader_election.cc:288] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:07:56.287199 1632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:56.287472 1632 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [term 1 LEADER]: Becoming Leader. State: Replica: b9b72457957e4349bf79eb081bc85c03, State: Running, Role: LEADER
I20260924 20:07:56.287770 1632 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b72457957e4349bf79eb081bc85c03" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46337 } }
I20260924 20:07:56.288297 1627 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:07:56.289606 1634 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b72457957e4349bf79eb081bc85c03. Latest consensus state: current_term: 1 leader_uuid: "b9b72457957e4349bf79eb081bc85c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b72457957e4349bf79eb081bc85c03" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46337 } } }
I20260924 20:07:56.289709 1634 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:56.290050 1633 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9b72457957e4349bf79eb081bc85c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b72457957e4349bf79eb081bc85c03" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46337 } } }
I20260924 20:07:56.290158 1633 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03 [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:56.290311 1642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:07:56.293408 1642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:07:56.298892 1642 catalog_manager.cc:1347] Generated new cluster ID: 9dbcc5fd71874c9fb849735a3864091a
I20260924 20:07:56.298964 1642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:07:56.312310 1642 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:07:56.313292 1642 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:07:56.325163 1642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03: Generated new TSK 0
I20260924 20:07:56.326114 1642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:07:56.363601 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46337
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:56.477996 1653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:56.478307 1653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:56.478394 1653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:56.482560 1653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:56.482645 1653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:56.482724 1653 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:07:56.482800 1653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:07:56.488129 1653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:56.489264 1653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:56.490329 1653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:56.499779 1659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:56.499779 1661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:56.499843 1658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:56.500046 1653 server_base.cc:1034] running on GCE node
I20260924 20:07:56.500463 1653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:56.501084 1653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:56.502259 1653 hybrid_clock.cc:648] HybridClock initialized: now 1790280476502241 us; error 31 us; skew 500 ppm
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:56.505098 1653 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:07:56.505982 1653 webserver.cc:460] Webserver started at http://127.0.115.129:37909/ using document root <none> and password file <none>
I20260924 20:07:56.506541 1653 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:56.506628 1653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:56.506870 1653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:56.509014 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "7cc79ed6cb7145639e94951912dd8cfa"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "d25d07e2399069c4ca60085760fdd2b3"
server_key_iv: "08f27c0db7cd11f7d83c7817b8aaa2d0"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.509636 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "7cc79ed6cb7145639e94951912dd8cfa"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "d25d07e2399069c4ca60085760fdd2b3"
server_key_iv: "08f27c0db7cd11f7d83c7817b8aaa2d0"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.513592 1653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:07:56.516129 1668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:56.517344 1653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:07:56.517472 1653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "7cc79ed6cb7145639e94951912dd8cfa"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "d25d07e2399069c4ca60085760fdd2b3"
server_key_iv: "08f27c0db7cd11f7d83c7817b8aaa2d0"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.517587 1653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:56.536460 1653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:56.539888 1653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:56.540114 1653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:56.541599 1653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:56.541678 1653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:56.541750 1653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:56.541783 1653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:56.553010 1653 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:45637
I20260924 20:07:56.553104 1783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:45637 every 8 connection(s)
I20260924 20:07:56.554116 1653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:07:56.557127 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.542701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:52169 (local address 127.0.115.190:46337)
0924 20:07:56.542839 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:56.542844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:56.544052 (+ 1208us) server_negotiation.cc:407] Connection header received
0924 20:07:56.545322 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:56.545326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:56.545390 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:56.545505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:56.546898 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.547498 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.548822 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.549160 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.549298 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.553130 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:56.553136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:56.553140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:56.553174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:56.555110 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.555680 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.555683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.555684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.555773 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.556152 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.556155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.556156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.556351 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:56.556432 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:56.556893 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 20:07:56.557004 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:07:56.557909 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.543288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46337 (local address 127.0.115.129:52169)
0924 20:07:56.543856 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:56.543904 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:56.545070 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:56.545779 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:56.545786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:56.546143 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:56.546736 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.546753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.547647 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.547651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.548685 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.548695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.549423 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.549428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.549602 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.550924 (+ 1322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:56.550949 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:56.552881 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:56.555223 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.555227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.555239 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.555488 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.555859 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.555861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.555863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.556060 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.556438 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:56.556442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:56.556678 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 20:07:56.556826 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 20:07:56.557081 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":130,"threads_started":1}
I20260924 20:07:56.559237 1784 heartbeater.cc:347] Connected to a master server at 127.0.115.190:46337
I20260924 20:07:56.559396 1784 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:56.559878 1784 heartbeater.cc:511] Master 127.0.115.190:46337 requested a full tablet report, sending...
I20260924 20:07:56.560410 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1653
I20260924 20:07:56.560535 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:07:56.560837 462 external_mini_cluster.cc:1428] Setting key f8772dc813ba43eee04a227d4ad7f899
I20260924 20:07:56.561705 1580 ts_manager.cc:194] Registered new tserver with Master: 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637)
I20260924 20:07:56.563337 1580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:52169
I20260924 20:07:56.569478 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.561556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:48505 (local address 127.0.115.190:46337)
0924 20:07:56.561675 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:56.561679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:56.561942 (+ 263us) server_negotiation.cc:407] Connection header received
0924 20:07:56.562130 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:56.562133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:56.562181 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:56.562275 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:56.562811 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.563325 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.564436 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.564743 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.564857 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.566309 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:56.566312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:56.566314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:56.566334 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:56.567762 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.568330 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.568332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.568333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.568405 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.568726 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.568728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.568729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.568884 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:56.568958 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:56.569274 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 20:07:56.569365 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:07:56.569623 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.561578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46337 (local address 127.0.115.129:48505)
0924 20:07:56.561861 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:56.561876 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:56.562005 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:56.562405 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:56.562407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:56.562594 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:56.562697 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.562702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.563509 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.563515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.564304 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.564320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.564874 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.564879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.564992 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.565678 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:56.565698 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:56.566172 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:56.567877 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.567884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.567888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.568227 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.568500 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.568505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.568510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.568632 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.568979 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:56.568984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:56.569099 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 20:07:56.569214 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 20:07:56.569377 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":8,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:07:56.589100 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46337
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:56.703517 1793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:56.703802 1793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:56.703876 1793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:56.707914 1793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:56.708034 1793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:56.708081 1793 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:07:56.708140 1793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:07:56.713663 1793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:56.714733 1793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:56.715761 1793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:56.725509 1799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:56.725512 1798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:56.725661 1793 server_base.cc:1034] running on GCE node
W20260924 20:07:56.725509 1801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:56.726112 1793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:56.726739 1793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:56.727922 1793 hybrid_clock.cc:648] HybridClock initialized: now 1790280476727900 us; error 29 us; skew 500 ppm
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:56.730670 1793 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:07:56.731530 1793 webserver.cc:460] Webserver started at http://127.0.115.130:36319/ using document root <none> and password file <none>
I20260924 20:07:56.732151 1793 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:56.732235 1793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:56.732460 1793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:56.734524 1793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "5c2cc9d6919cb7e9562a9597aa6f0db0"
server_key_iv: "d38bc4ed2c4be856947bb0acd665502d"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.735092 1793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "5c2cc9d6919cb7e9562a9597aa6f0db0"
server_key_iv: "d38bc4ed2c4be856947bb0acd665502d"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.738952 1793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:07:56.741374 1808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:56.742525 1793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:07:56.742663 1793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "5c2cc9d6919cb7e9562a9597aa6f0db0"
server_key_iv: "d38bc4ed2c4be856947bb0acd665502d"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.742770 1793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:56.765182 1793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:56.768020 1793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:56.768224 1793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:56.769752 1793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:56.769819 1793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:56.769887 1793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:56.769932 1793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:56.780838 1793 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:41197
I20260924 20:07:56.780858 1923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:41197 every 8 connection(s)
I20260924 20:07:56.781901 1793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:07:56.783423 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.770527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:51713 (local address 127.0.115.190:46337)
0924 20:07:56.770698 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:56.770704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:56.771685 (+ 981us) server_negotiation.cc:407] Connection header received
0924 20:07:56.772511 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:56.772514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:56.772569 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:56.772670 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:56.774076 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.774870 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.776120 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.776538 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.776702 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.779424 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:56.779429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:56.779431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:56.779462 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:56.781171 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.781759 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.781761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.781763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.781842 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.782191 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.782193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.782194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.782453 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:56.782548 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:56.783126 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 20:07:56.783254 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:07:56.784241 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.770929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46337 (local address 127.0.115.130:51713)
0924 20:07:56.771373 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:56.771423 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:56.772299 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:56.772864 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:56.772873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:56.773369 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:56.773934 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.773947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.775004 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.775008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.775982 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.775998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.776751 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.776770 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.776951 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.777583 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:56.777601 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:56.779194 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:56.781297 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.781302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.781316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.781665 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.781931 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.781937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.781939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.782102 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.782602 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:56.782611 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:56.782895 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 20:07:56.783081 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 20:07:56.783401 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":119,"threads_started":1}
I20260924 20:07:56.785455 1924 heartbeater.cc:347] Connected to a master server at 127.0.115.190:46337
I20260924 20:07:56.785485 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1793
I20260924 20:07:56.785589 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:07:56.785619 1924 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:56.785851 462 external_mini_cluster.cc:1428] Setting key 7606e3fcbbb69dc37c00bfbd8045279a
I20260924 20:07:56.786181 1924 heartbeater.cc:511] Master 127.0.115.190:46337 requested a full tablet report, sending...
I20260924 20:07:56.787487 1581 ts_manager.cc:194] Registered new tserver with Master: 6e98a2dbf7a44a07a16c612b5b62ff9f (127.0.115.130:41197)
I20260924 20:07:56.788389 1581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:51713
I20260924 20:07:56.796216 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.787883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:36161 (local address 127.0.115.190:46337)
0924 20:07:56.788741 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:56.788747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:56.788763 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:07:56.788803 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:56.788808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:56.788858 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:56.788964 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:56.789693 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.790545 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.791359 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.791564 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.791640 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.792907 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:56.792910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:56.792912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:56.792933 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:56.794397 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.794960 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.794963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.794964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.795037 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:56.795359 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:56.795361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:56.795362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:56.795557 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:56.795667 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:56.796002 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 20:07:56.796100 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762}
I20260924 20:07:56.796339 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.787629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46337 (local address 127.0.115.130:36161)
0924 20:07:56.787869 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:56.787883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:56.788009 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:56.789325 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:56.789327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:56.789484 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:56.789584 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.789589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.790655 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.790658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.791260 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.791269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.791679 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.791684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.791807 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.792304 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:56.792323 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:56.792766 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:56.794563 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.794569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.794573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.794857 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.795135 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:56.795141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:56.795146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:56.795265 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:56.795685 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:56.795688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:56.795791 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 20:07:56.795912 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:07:56.796094 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:07:56.815685 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46337
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:56.925817 1933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:56.926100 1933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:56.926172 1933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:56.930313 1933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:56.930389 1933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:56.930455 1933 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:07:56.930528 1933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:07:56.935690 1933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:56.936810 1933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:56.937873 1933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:56.947768 1938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:56.947814 1939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:56.947855 1933 server_base.cc:1034] running on GCE node
W20260924 20:07:56.947786 1941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:56.948446 1933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:56.949158 1933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:56.950338 1933 hybrid_clock.cc:648] HybridClock initialized: now 1790280476950307 us; error 44 us; skew 500 ppm
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:56.953126 1933 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:07:56.954128 1933 webserver.cc:460] Webserver started at http://127.0.115.131:36151/ using document root <none> and password file <none>
I20260924 20:07:56.954717 1933 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:56.954806 1933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:56.955034 1933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:56.957212 1933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "cd1f0aa6952348d683881f91743027b5"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "37d13a59119e4f9299f0df16e83e5832"
server_key_iv: "86b0b6a265b85bbf4685fe0a69d67bff"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.957791 1933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "cd1f0aa6952348d683881f91743027b5"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "37d13a59119e4f9299f0df16e83e5832"
server_key_iv: "86b0b6a265b85bbf4685fe0a69d67bff"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.961831 1933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 20:07:56.964421 1948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:56.965664 1933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:07:56.965819 1933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "cd1f0aa6952348d683881f91743027b5"
format_stamp: "Formatted at 2026-09-24 20:07:56 on dist-test-slave-2pm7"
server_key: "37d13a59119e4f9299f0df16e83e5832"
server_key_iv: "86b0b6a265b85bbf4685fe0a69d67bff"
server_key_version: "encryptionkey@0"
I20260924 20:07:56.965946 1933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:56.982962 1933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:56.986143 1933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:56.986356 1933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:56.988070 1933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:56.988134 1933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:56.988201 1933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:56.988238 1933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:07:56 dist-test-slave-2pm7 krb5kdc[1536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:56.999747 1933 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:42977
I20260924 20:07:57.000625 2063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:42977 every 8 connection(s)
I20260924 20:07:57.000941 1933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:07:57.001920 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 1933
I20260924 20:07:57.002123 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:07:57.002508 462 external_mini_cluster.cc:1428] Setting key 1dfb10733bb465b8b3daf53cc2147218
I20260924 20:07:57.002789 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.988831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:53711 (local address 127.0.115.190:46337)
0924 20:07:56.988980 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:56.988985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:56.990034 (+ 1049us) server_negotiation.cc:407] Connection header received
0924 20:07:56.991390 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:56.991393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:56.991469 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:56.991563 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:56.992972 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.993792 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.995310 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.995621 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.995738 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.998701 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:56.998705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:56.998707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:56.998740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:57.000411 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.001026 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.001030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.001031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.001120 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.001551 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.001555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.001557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.001817 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:57.001927 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.002498 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.002626 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:07:57.003755 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:56.989328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46337 (local address 127.0.115.131:53711)
0924 20:07:56.989858 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:56.989921 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:56.991098 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:56.991863 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:56.991870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:56.992251 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:56.992816 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.992837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.993947 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.993950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.995186 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:56.995196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:56.995788 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:56.995792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:56.996023 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:56.996629 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:56.996648 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:56.998482 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:57.000556 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:57.000563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:57.000576 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:57.000920 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:57.001224 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:57.001226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:57.001227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:57.001425 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:57.001920 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:57.001925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:57.002242 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.002450 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.002785 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260924 20:07:57.005028 2064 heartbeater.cc:347] Connected to a master server at 127.0.115.190:46337
I20260924 20:07:57.005165 2064 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:57.005796 2064 heartbeater.cc:511] Master 127.0.115.190:46337 requested a full tablet report, sending...
I20260924 20:07:57.006843 1580 ts_manager.cc:194] Registered new tserver with Master: cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977)
I20260924 20:07:57.007609 1580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:53711
I20260924 20:07:57.014122 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.007342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46337 (local address 127.0.115.131:51929)
0924 20:07:57.007664 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.007675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.007788 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.008061 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.008064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.008272 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:57.008395 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.008403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.009129 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.009132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.009903 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.009912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.010299 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.010301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.010372 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.010789 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:57.010800 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:57.011136 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:57.012814 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:57.012817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:57.012818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:57.012996 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:57.013254 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:57.013257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:57.013259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:57.013343 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:57.013720 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:57.013722 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:57.013800 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.013884 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.013988 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":94,"threads_started":1}
I20260924 20:07:57.014127 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.007411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:51929 (local address 127.0.115.190:46337)
0924 20:07:57.007535 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.007538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.007749 (+ 211us) server_negotiation.cc:407] Connection header received
0924 20:07:57.007883 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.007885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.007932 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.008031 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:57.008517 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.009008 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.010001 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.010202 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.010284 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.011255 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:57.011258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:57.011260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:57.011281 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:57.012711 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.013084 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.013087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.013088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.013157 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.013426 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.013427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.013429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.013638 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:57.013715 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.013940 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.014024 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:07:57.016419 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:07:57 dist-test-slave-2pm7 krb5kdc[1536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280477, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:07:57 dist-test-slave-2pm7 krb5kdc[1536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280477, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:57.040669 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.032953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.0.115.190:46337)
0924 20:07:57.033071 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.033076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.033093 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:07:57.033131 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.033133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.033171 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.033243 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:57.033925 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.034430 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.035233 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.035437 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.035509 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.037899 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:57.037902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:57.037904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:57.037927 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:57.039299 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.039648 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.039651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.039652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.039724 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.039934 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.039936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.039937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.040146 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:57.040242 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.040451 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.040541 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 20:07:57.043431 1580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:07:57.046631 1580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:07:57.061343 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.055250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.0.115.130:41197)
0924 20:07:57.056240 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.056253 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.056278 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:07:57.056360 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.056366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.056591 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.056761 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:57.057242 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.058111 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.059814 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.060671 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.060797 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.060967 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.061095 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.061185 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":122,"threads_started":1}
I20260924 20:07:57.061368 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.055152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:45637 (local address 127.0.0.1:35546)
0924 20:07:57.055722 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.055780 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.055918 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.057073 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.057076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.057093 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:57.057203 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.057212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.058671 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.058675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.059842 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.059849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.060929 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.060933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.061028 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.061055 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.061110 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.061203 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":49,"spinlock_wait_cycles":26368,"thread_start_us":104,"threads_started":1}
I20260924 20:07:57.061754 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.055910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:42977 (local address 127.0.0.1:35106)
0924 20:07:57.056481 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.056494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.056580 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.057305 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.057310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.057333 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:57.057422 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.057428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.058737 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.058742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.060194 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.060205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.061365 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.061369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.061464 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.061488 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.061555 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.061619 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":24,"thread_start_us":369,"threads_started":1}
I20260924 20:07:57.061338 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.055127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:41197 (local address 127.0.0.1:45508)
0924 20:07:57.055722 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.055780 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.055918 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.056932 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.056938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.056963 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:57.057108 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.057115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.058222 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.058227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.059688 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.059699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.060796 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.060800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.060906 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.060943 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.061064 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.061186 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"spinlock_wait_cycles":7040,"thread_start_us":105,"threads_started":1}
I20260924 20:07:57.061978 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.056008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.0.115.129:45637)
0924 20:07:57.056627 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.056642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.056667 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:07:57.056741 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.056746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.056917 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.057063 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:57.057351 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.058549 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.059968 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.060785 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.061521 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.061676 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.061725 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.061823 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":97,"threads_started":1}
I20260924 20:07:57.062283 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.056201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.115.131:42977)
0924 20:07:57.056854 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.056867 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.056889 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:07:57.056968 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.056973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.057161 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.057324 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:57.057535 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.058652 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.060317 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.061063 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.061158 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.061306 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.062048 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.062150 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":111,"threads_started":1}
I20260924 20:07:57.064675 1840 tablet_service.cc:1467] Processing CreateTablet for tablet 3f321bbbef634d01bee625012e1a6f79 (DEFAULT_TABLE table=test-table [id=c6a4940561d346679c3be6c2055fcf2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:57.064796 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 3f321bbbef634d01bee625012e1a6f79 (DEFAULT_TABLE table=test-table [id=c6a4940561d346679c3be6c2055fcf2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:57.064888 1991 tablet_service.cc:1467] Processing CreateTablet for tablet 3f321bbbef634d01bee625012e1a6f79 (DEFAULT_TABLE table=test-table [id=c6a4940561d346679c3be6c2055fcf2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:57.065569 1840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f321bbbef634d01bee625012e1a6f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.065786 1991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f321bbbef634d01bee625012e1a6f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.065786 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f321bbbef634d01bee625012e1a6f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.072850 2087 tablet_bootstrap.cc:492] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Bootstrap starting.
I20260924 20:07:57.073930 2088 tablet_bootstrap.cc:492] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Bootstrap starting.
I20260924 20:07:57.075629 2087 tablet_bootstrap.cc:654] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.075974 2088 tablet_bootstrap.cc:654] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.076128 2089 tablet_bootstrap.cc:492] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Bootstrap starting.
I20260924 20:07:57.076596 2087 log.cc:824] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:57.077119 2088 log.cc:824] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:57.078215 2089 tablet_bootstrap.cc:654] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.078621 2087 tablet_bootstrap.cc:492] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: No bootstrap required, opened a new log
I20260924 20:07:57.078823 2087 ts_tablet_manager.cc:1397] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:07:57.078969 2089 log.cc:824] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:57.080888 2089 tablet_bootstrap.cc:492] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: No bootstrap required, opened a new log
I20260924 20:07:57.080888 2088 tablet_bootstrap.cc:492] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: No bootstrap required, opened a new log
I20260924 20:07:57.081070 2089 ts_tablet_manager.cc:1397] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:07:57.081070 2088 ts_tablet_manager.cc:1397] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:07:57.082057 2087 raft_consensus.cc:348] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.082299 2087 raft_consensus.cc:374] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.082352 2087 raft_consensus.cc:729] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e98a2dbf7a44a07a16c612b5b62ff9f, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.082799 2087 consensus_queue.cc:260] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.083434 1924 heartbeater.cc:503] Master 127.0.115.190:46337 was elected leader, sending a full tablet report...
I20260924 20:07:57.083803 2087 ts_tablet_manager.cc:1428] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:07:57.085014 2089 raft_consensus.cc:348] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.085278 2089 raft_consensus.cc:374] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.085341 2089 raft_consensus.cc:729] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1f0aa6952348d683881f91743027b5, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.085263 2088 raft_consensus.cc:348] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.085495 2088 raft_consensus.cc:374] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.085549 2088 raft_consensus.cc:729] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc79ed6cb7145639e94951912dd8cfa, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.085798 2089 consensus_queue.cc:260] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.085894 2088 consensus_queue.cc:260] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.086503 2064 heartbeater.cc:503] Master 127.0.115.190:46337 was elected leader, sending a full tablet report...
I20260924 20:07:57.086818 2089 ts_tablet_manager.cc:1428] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:07:57.086818 2088 ts_tablet_manager.cc:1428] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:07:57.087139 1784 heartbeater.cc:503] Master 127.0.115.190:46337 was elected leader, sending a full tablet report...
I20260924 20:07:57.090937 2094 raft_consensus.cc:482] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:07:57.091059 2094 raft_consensus.cc:504] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.092248 2094 leader_election.cc:288] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e98a2dbf7a44a07a16c612b5b62ff9f (127.0.115.130:41197), 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637)
I20260924 20:07:57.096091 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.092438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:41197 (local address 127.0.115.131:53551)
0924 20:07:57.092592 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.092608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.092709 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.093012 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.093016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.093036 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:57.093174 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.093182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.093973 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.093980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.094788 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.094797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.095632 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.095635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.095767 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.095792 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.095862 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.095933 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 20:07:57.096101 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.092477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:53551 (local address 127.0.115.130:41197)
0924 20:07:57.092587 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.092591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.092802 (+ 211us) server_negotiation.cc:407] Connection header received
0924 20:07:57.092853 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.092857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.092912 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.092985 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:57.093287 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.093865 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.094910 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.095517 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.095609 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.095648 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.095897 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.095998 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 20:07:57.096850 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.092510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:44149 (local address 127.0.115.129:45637)
0924 20:07:57.092636 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.092640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.092929 (+ 289us) server_negotiation.cc:407] Connection header received
0924 20:07:57.093022 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.093024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.093066 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.093139 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:57.093424 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.093938 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.095523 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.096155 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.096273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.096308 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.096676 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.096752 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:07:57.096856 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.092457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:45637 (local address 127.0.115.131:44149)
0924 20:07:57.092844 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.092856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.092944 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.093168 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.093170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.093184 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:57.093319 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.093326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.094042 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.094044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.095393 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.095404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.096369 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.096372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.096491 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.096512 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.096655 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.096726 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260924 20:07:57.096822 1870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f321bbbef634d01bee625012e1a6f79" candidate_uuid: "cd1f0aa6952348d683881f91743027b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" is_pre_election: true
I20260924 20:07:57.096983 2093 raft_consensus.cc:482] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:07:57.097103 2093 raft_consensus.cc:504] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.097436 1738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f321bbbef634d01bee625012e1a6f79" candidate_uuid: "cd1f0aa6952348d683881f91743027b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc79ed6cb7145639e94951912dd8cfa" is_pre_election: true
I20260924 20:07:57.097506 1870 raft_consensus.cc:2454] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1f0aa6952348d683881f91743027b5 in term 0.
I20260924 20:07:57.097699 1738 raft_consensus.cc:2454] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1f0aa6952348d683881f91743027b5 in term 0.
I20260924 20:07:57.098004 1950 leader_election.cc:302] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e98a2dbf7a44a07a16c612b5b62ff9f, cd1f0aa6952348d683881f91743027b5; no voters:
I20260924 20:07:57.098140 2093 leader_election.cc:288] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637), cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977)
I20260924 20:07:57.098430 2094 raft_consensus.cc:2781] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:07:57.098508 2094 raft_consensus.cc:482] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:07:57.098551 2094 raft_consensus.cc:3037] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.100095 2094 raft_consensus.cc:504] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.100714 1870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f321bbbef634d01bee625012e1a6f79" candidate_uuid: "cd1f0aa6952348d683881f91743027b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f"
I20260924 20:07:57.100832 1870 raft_consensus.cc:3037] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.100927 1738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f321bbbef634d01bee625012e1a6f79" candidate_uuid: "cd1f0aa6952348d683881f91743027b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc79ed6cb7145639e94951912dd8cfa"
I20260924 20:07:57.101058 1738 raft_consensus.cc:3037] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.102138 1870 raft_consensus.cc:2454] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1f0aa6952348d683881f91743027b5 in term 1.
I20260924 20:07:57.102442 1738 raft_consensus.cc:2454] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1f0aa6952348d683881f91743027b5 in term 1.
I20260924 20:07:57.102924 1950 leader_election.cc:302] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e98a2dbf7a44a07a16c612b5b62ff9f, cd1f0aa6952348d683881f91743027b5; no voters:
I20260924 20:07:57.102977 2094 leader_election.cc:288] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [CANDIDATE]: Term 1 election: Requested vote from peers 6e98a2dbf7a44a07a16c612b5b62ff9f (127.0.115.130:41197), 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637)
I20260924 20:07:57.103168 2094 raft_consensus.cc:2781] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:57.103238 2094 raft_consensus.cc:686] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 1 LEADER]: Becoming Leader. State: Replica: cd1f0aa6952348d683881f91743027b5, State: Running, Role: LEADER
I20260924 20:07:57.103376 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.098280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:45637 (local address 127.0.115.130:34465)
0924 20:07:57.098437 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.098452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.098528 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.099102 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.099104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.099122 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:57.099246 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.099252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.100192 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.100198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.101444 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.101457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.102872 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.102876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.102995 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.103022 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.103103 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.103227 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":21}
I20260924 20:07:57.103578 2094 consensus_queue.cc:237] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.104022 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.098702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:34465 (local address 127.0.115.129:45637)
0924 20:07:57.098811 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.098815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.098829 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:07:57.098872 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.098876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.098927 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.099007 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:57.099360 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.100067 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.101576 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.102354 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.102622 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.102695 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.103804 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.103882 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:07:57.104449 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.098451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:36203 (local address 127.0.115.131:42977)
0924 20:07:57.098832 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.098838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.098852 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:07:57.099000 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.099003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.099043 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.099085 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:07:57.100034 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.100688 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.103000 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.103710 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.103810 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.103860 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.104249 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.104332 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260924 20:07:57.104460 1738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f321bbbef634d01bee625012e1a6f79" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc79ed6cb7145639e94951912dd8cfa" is_pre_election: true
I20260924 20:07:57.104732 1738 raft_consensus.cc:2379] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in current term 1: Already voted for candidate cd1f0aa6952348d683881f91743027b5 in this term.
I20260924 20:07:57.104875 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.098407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:42977 (local address 127.0.115.130:36203)
0924 20:07:57.098776 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.098792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.098881 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.099199 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.099202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.099219 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:07:57.099833 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.099852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.100847 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.100850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.102842 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.102855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.104003 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.104006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.104102 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.104125 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.104657 (+ 532us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.104734 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":28,"thread_start_us":119,"threads_started":1}
I20260924 20:07:57.105444 2018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f321bbbef634d01bee625012e1a6f79" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1f0aa6952348d683881f91743027b5" is_pre_election: true
I20260924 20:07:57.106148 1810 leader_election.cc:302] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e98a2dbf7a44a07a16c612b5b62ff9f; no voters: 7cc79ed6cb7145639e94951912dd8cfa, cd1f0aa6952348d683881f91743027b5
I20260924 20:07:57.106760 2093 raft_consensus.cc:2726] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 20:07:57.107612 1579 catalog_manager.cc:5495] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd1f0aa6952348d683881f91743027b5 (127.0.115.131). New cstate: current_term: 1 leader_uuid: "cd1f0aa6952348d683881f91743027b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } health_report { overall_health: HEALTHY } } }
I20260924 20:07:57.123595 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.119820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.0.115.131:42977)
0924 20:07:57.119970 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.119978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.119994 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:07:57.120207 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.120209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.120263 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.120346 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:07:57.120629 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.121176 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.122142 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.122411 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.122491 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.122774 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:07:57.123178 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:07:57.123279 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.123437 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.123487 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:07:57.129420 1738 raft_consensus.cc:1261] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Refusing update from remote peer cd1f0aa6952348d683881f91743027b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:07:57.129483 1870 raft_consensus.cc:1261] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Refusing update from remote peer cd1f0aa6952348d683881f91743027b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:07:57.130062 2098 consensus_queue.cc:1035] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:07:57.130308 2094 consensus_queue.cc:1035] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:07:57.138494 2102 mvcc.cc:205] Tried to move back new op lower bound from 7332988834314522624 to 7332988834220044288. Current Snapshot: MvccSnapshot[applied={T|T < 7332988834314522624}]
I20260924 20:07:57.148082 1579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:07:57.148274 1579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:07:57.150763 1579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03: Sending DeleteTablet for 3 replicas of tablet 3f321bbbef634d01bee625012e1a6f79
I20260924 20:07:57.151633 1991 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f321bbbef634d01bee625012e1a6f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:07:57 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:35106
I20260924 20:07:57.151739 1840 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f321bbbef634d01bee625012e1a6f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:07:57 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:45508
I20260924 20:07:57.152149 1718 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f321bbbef634d01bee625012e1a6f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:07:57 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:35546
I20260924 20:07:57.152436 2112 tablet_replica.cc:330] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: stopping tablet replica
I20260924 20:07:57.152654 1579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:07:57.152792 2112 raft_consensus.cc:2229] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 20:07:57.153074 1579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:07:57.153088 2112 raft_consensus.cc:2258] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:07:57.153488 2111 tablet_replica.cc:330] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: stopping tablet replica
I20260924 20:07:57.153780 2111 raft_consensus.cc:2229] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:07:57.154150 2111 raft_consensus.cc:2258] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:07:57.155686 2112 ts_tablet_manager.cc:1905] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:07:57.158141 1718 tablet_service.cc:1467] Processing CreateTablet for tablet f235db5965894da3bb0d050e978b6b2e (DEFAULT_TABLE table=test-table [id=6ed73099f5ee4dfa8cda1b8be5a6fd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:57.158250 1840 tablet_service.cc:1467] Processing CreateTablet for tablet f235db5965894da3bb0d050e978b6b2e (DEFAULT_TABLE table=test-table [id=6ed73099f5ee4dfa8cda1b8be5a6fd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:57.158427 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f235db5965894da3bb0d050e978b6b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.158533 1840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f235db5965894da3bb0d050e978b6b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.159929 2111 ts_tablet_manager.cc:1905] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:07:57.160735 2087 tablet_bootstrap.cc:492] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Bootstrap starting.
I20260924 20:07:57.160851 2088 tablet_bootstrap.cc:492] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Bootstrap starting.
I20260924 20:07:57.161357 1991 tablet_service.cc:1467] Processing CreateTablet for tablet f235db5965894da3bb0d050e978b6b2e (DEFAULT_TABLE table=test-table [id=6ed73099f5ee4dfa8cda1b8be5a6fd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:07:57.161612 1991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f235db5965894da3bb0d050e978b6b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.161679 2087 tablet_bootstrap.cc:654] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.161804 2088 tablet_bootstrap.cc:654] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.162812 2087 tablet_bootstrap.cc:492] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: No bootstrap required, opened a new log
I20260924 20:07:57.162916 2087 ts_tablet_manager.cc:1397] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:07:57.163307 2087 raft_consensus.cc:348] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.163424 2087 raft_consensus.cc:374] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.163463 2087 raft_consensus.cc:729] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e98a2dbf7a44a07a16c612b5b62ff9f, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.163589 2087 consensus_queue.cc:260] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.163844 2087 ts_tablet_manager.cc:1428] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:07:57.163981 2111 ts_tablet_manager.cc:1918] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:07:57.164083 2111 log.cc:1196] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/3f321bbbef634d01bee625012e1a6f79
I20260924 20:07:57.164156 2089 tablet_bootstrap.cc:492] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Bootstrap starting.
I20260924 20:07:57.164433 2111 ts_tablet_manager.cc:1939] T 3f321bbbef634d01bee625012e1a6f79 P cd1f0aa6952348d683881f91743027b5: Deleting consensus metadata
I20260924 20:07:57.165081 2089 tablet_bootstrap.cc:654] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.165664 1566 catalog_manager.cc:4841] TS cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977): tablet 3f321bbbef634d01bee625012e1a6f79 (table test-table [id=c6a4940561d346679c3be6c2055fcf2d]) successfully deleted
I20260924 20:07:57.158542 2112 ts_tablet_manager.cc:1918] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:07:57.165957 2112 log.cc:1196] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/3f321bbbef634d01bee625012e1a6f79
I20260924 20:07:57.166296 2112 ts_tablet_manager.cc:1939] T 3f321bbbef634d01bee625012e1a6f79 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Deleting consensus metadata
I20260924 20:07:57.166615 2089 tablet_bootstrap.cc:492] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: No bootstrap required, opened a new log
I20260924 20:07:57.166693 2089 ts_tablet_manager.cc:1397] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:07:57.166635 2113 tablet_replica.cc:330] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: stopping tablet replica
I20260924 20:07:57.166999 2113 raft_consensus.cc:2229] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:07:57.167128 2089 raft_consensus.cc:348] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.167243 2089 raft_consensus.cc:374] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.167284 2089 raft_consensus.cc:729] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1f0aa6952348d683881f91743027b5, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.167346 2113 raft_consensus.cc:2258] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:07:57.167415 2089 consensus_queue.cc:260] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.167483 1566 catalog_manager.cc:4841] TS 6e98a2dbf7a44a07a16c612b5b62ff9f (127.0.115.130:41197): tablet 3f321bbbef634d01bee625012e1a6f79 (table test-table [id=c6a4940561d346679c3be6c2055fcf2d]) successfully deleted
I20260924 20:07:57.167701 2089 ts_tablet_manager.cc:1428] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:07:57.168473 2088 tablet_bootstrap.cc:492] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: No bootstrap required, opened a new log
I20260924 20:07:57.168573 2088 ts_tablet_manager.cc:1397] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 20:07:57.169049 2088 raft_consensus.cc:348] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.169171 2088 raft_consensus.cc:374] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.169212 2088 raft_consensus.cc:729] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc79ed6cb7145639e94951912dd8cfa, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.169346 2088 consensus_queue.cc:260] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.169404 2113 ts_tablet_manager.cc:1905] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:07:57.169600 2088 ts_tablet_manager.cc:1428] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:07:57.172765 2113 ts_tablet_manager.cc:1918] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:07:57.172866 2113 log.cc:1196] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/3f321bbbef634d01bee625012e1a6f79
I20260924 20:07:57.173159 2113 ts_tablet_manager.cc:1939] T 3f321bbbef634d01bee625012e1a6f79 P 7cc79ed6cb7145639e94951912dd8cfa: Deleting consensus metadata
I20260924 20:07:57.174266 1565 catalog_manager.cc:4841] TS 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637): tablet 3f321bbbef634d01bee625012e1a6f79 (table test-table [id=c6a4940561d346679c3be6c2055fcf2d]) successfully deleted
W20260924 20:07:57.252043 2065 tablet.cc:2369] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:07:57.283221 1925 tablet.cc:2369] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:07:57.305691 1785 tablet.cc:2369] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:07:57.475117 2093 raft_consensus.cc:482] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:07:57.475268 2093 raft_consensus.cc:504] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.475665 2093 leader_election.cc:288] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637), cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977)
I20260924 20:07:57.476274 1738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f235db5965894da3bb0d050e978b6b2e" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc79ed6cb7145639e94951912dd8cfa" is_pre_election: true
I20260924 20:07:57.476274 2018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f235db5965894da3bb0d050e978b6b2e" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1f0aa6952348d683881f91743027b5" is_pre_election: true
I20260924 20:07:57.476450 1738 raft_consensus.cc:2454] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 0.
I20260924 20:07:57.476511 2018 raft_consensus.cc:2454] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 0.
I20260924 20:07:57.476925 1810 leader_election.cc:302] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e98a2dbf7a44a07a16c612b5b62ff9f, 7cc79ed6cb7145639e94951912dd8cfa, cd1f0aa6952348d683881f91743027b5; no voters:
I20260924 20:07:57.477120 2093 raft_consensus.cc:2781] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:07:57.477193 2093 raft_consensus.cc:482] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:07:57.477245 2093 raft_consensus.cc:3037] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.478266 2093 raft_consensus.cc:504] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.478675 2093 leader_election.cc:288] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 election: Requested vote from peers 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637), cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977)
I20260924 20:07:57.479050 1738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f235db5965894da3bb0d050e978b6b2e" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc79ed6cb7145639e94951912dd8cfa"
I20260924 20:07:57.479084 2018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f235db5965894da3bb0d050e978b6b2e" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1f0aa6952348d683881f91743027b5"
I20260924 20:07:57.479192 1738 raft_consensus.cc:3037] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.479223 2018 raft_consensus.cc:3037] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.480347 2018 raft_consensus.cc:2454] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 1.
I20260924 20:07:57.480422 1738 raft_consensus.cc:2454] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 1.
I20260924 20:07:57.480641 1810 leader_election.cc:302] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e98a2dbf7a44a07a16c612b5b62ff9f, cd1f0aa6952348d683881f91743027b5; no voters:
I20260924 20:07:57.480818 2093 raft_consensus.cc:2781] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:57.480981 2093 raft_consensus.cc:686] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 LEADER]: Becoming Leader. State: Replica: 6e98a2dbf7a44a07a16c612b5b62ff9f, State: Running, Role: LEADER
I20260924 20:07:57.481263 2093 consensus_queue.cc:237] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.483917 1580 catalog_manager.cc:5495] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e98a2dbf7a44a07a16c612b5b62ff9f (127.0.115.130). New cstate: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:07:57 dist-test-slave-2pm7 krb5kdc[1536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:57.555187 2125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.545832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46337 (local address 127.0.0.1:37110)
0924 20:07:57.546175 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.546189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.546293 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.546585 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.546588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.547057 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:57.547197 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.547204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.547973 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.547975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.548567 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.548575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.549020 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.549023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.549086 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.549668 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:57.549688 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:57.551417 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:57.553388 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:57.553398 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:57.553403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:57.553685 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:57.553991 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:57.553994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:57.553995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:57.554098 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:57.554565 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:57.554572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:57.554688 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.554807 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.555029 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":129,"threads_started":1}
I20260924 20:07:57.555327 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.545885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.0.115.190:46337)
0924 20:07:57.546164 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.546178 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.546261 (+ 83us) server_negotiation.cc:407] Connection header received
0924 20:07:57.546375 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.546377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.546420 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.546514 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:57.547305 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.547855 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.548677 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.548914 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.548992 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.551557 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:57.551561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:57.551563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:57.551584 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:57.553259 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.553788 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.553792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.553794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.553890 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.554193 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.554196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.554197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.554435 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:57.554533 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.554873 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.555125 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260924 20:07:57.559675 1580 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:37110:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:07:57.560478 1580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:07:57.565407 1718 tablet_service.cc:1467] Processing CreateTablet for tablet fad39498b2a7486a886e4612204953c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bc64382b9f54a04b814ee4217f8fba0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:07:57.565795 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad39498b2a7486a886e4612204953c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.569005 1991 tablet_service.cc:1467] Processing CreateTablet for tablet fad39498b2a7486a886e4612204953c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bc64382b9f54a04b814ee4217f8fba0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:07:57.569425 1991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad39498b2a7486a886e4612204953c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.571753 2089 tablet_bootstrap.cc:492] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: Bootstrap starting.
I20260924 20:07:57.572161 1840 tablet_service.cc:1467] Processing CreateTablet for tablet fad39498b2a7486a886e4612204953c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bc64382b9f54a04b814ee4217f8fba0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:07:57.572515 1840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad39498b2a7486a886e4612204953c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:57.572784 2089 tablet_bootstrap.cc:654] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.574141 2089 tablet_bootstrap.cc:492] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: No bootstrap required, opened a new log
I20260924 20:07:57.574244 2089 ts_tablet_manager.cc:1397] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:07:57.574690 2089 raft_consensus.cc:348] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.574827 2089 raft_consensus.cc:374] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.574874 2089 raft_consensus.cc:729] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1f0aa6952348d683881f91743027b5, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.575008 2089 consensus_queue.cc:260] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.575299 2089 ts_tablet_manager.cc:1428] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:07:57.575279 2106 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.575440 2106 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:07:57.576066 2087 tablet_bootstrap.cc:492] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Bootstrap starting.
I20260924 20:07:57.576135 2088 tablet_bootstrap.cc:492] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: Bootstrap starting.
I20260924 20:07:57.577124 2087 tablet_bootstrap.cc:654] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.577124 2088 tablet_bootstrap.cc:654] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: Neither blocks nor log segments found. Creating new log.
I20260924 20:07:57.578794 2088 tablet_bootstrap.cc:492] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: No bootstrap required, opened a new log
I20260924 20:07:57.578891 2087 tablet_bootstrap.cc:492] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: No bootstrap required, opened a new log
I20260924 20:07:57.578894 2088 ts_tablet_manager.cc:1397] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:07:57.578959 2087 ts_tablet_manager.cc:1397] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 20:07:57.579368 2087 raft_consensus.cc:348] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.579368 2088 raft_consensus.cc:348] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.579491 2087 raft_consensus.cc:374] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.579491 2088 raft_consensus.cc:374] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:57.579527 2087 raft_consensus.cc:729] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e98a2dbf7a44a07a16c612b5b62ff9f, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.579527 2088 raft_consensus.cc:729] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc79ed6cb7145639e94951912dd8cfa, State: Initialized, Role: FOLLOWER
I20260924 20:07:57.579659 2088 consensus_queue.cc:260] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.579659 2087 consensus_queue.cc:260] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.579975 2087 ts_tablet_manager.cc:1428] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:07:57.579975 2088 ts_tablet_manager.cc:1428] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:07:57.580000 2095 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.580175 2095 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:07:57.580366 2093 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.580509 2093 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:07:57.838706 2093 raft_consensus.cc:482] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:07:57.838855 2093 raft_consensus.cc:504] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.839392 2093 leader_election.cc:288] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637), cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977)
I20260924 20:07:57.839936 1738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad39498b2a7486a886e4612204953c4" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc79ed6cb7145639e94951912dd8cfa" is_pre_election: true
I20260924 20:07:57.840020 2018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad39498b2a7486a886e4612204953c4" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1f0aa6952348d683881f91743027b5" is_pre_election: true
I20260924 20:07:57.840145 1738 raft_consensus.cc:2454] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 0.
I20260924 20:07:57.840159 2018 raft_consensus.cc:2454] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 0.
I20260924 20:07:57.840484 1809 leader_election.cc:302] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e98a2dbf7a44a07a16c612b5b62ff9f, 7cc79ed6cb7145639e94951912dd8cfa; no voters:
I20260924 20:07:57.840734 2093 raft_consensus.cc:2781] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:07:57.840792 2093 raft_consensus.cc:482] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:07:57.840847 2093 raft_consensus.cc:3037] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.841799 2093 raft_consensus.cc:504] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.842129 2093 leader_election.cc:288] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 election: Requested vote from peers 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637), cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977)
I20260924 20:07:57.842500 1738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad39498b2a7486a886e4612204953c4" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc79ed6cb7145639e94951912dd8cfa"
I20260924 20:07:57.842520 2018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad39498b2a7486a886e4612204953c4" candidate_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd1f0aa6952348d683881f91743027b5"
I20260924 20:07:57.842653 1738 raft_consensus.cc:3037] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.842684 2018 raft_consensus.cc:3037] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:57.843564 2018 raft_consensus.cc:2454] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 1.
I20260924 20:07:57.843557 1738 raft_consensus.cc:2454] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e98a2dbf7a44a07a16c612b5b62ff9f in term 1.
I20260924 20:07:57.843871 1810 leader_election.cc:302] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e98a2dbf7a44a07a16c612b5b62ff9f, cd1f0aa6952348d683881f91743027b5; no voters:
I20260924 20:07:57.844064 2093 raft_consensus.cc:2781] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:57.844161 2093 raft_consensus.cc:686] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 LEADER]: Becoming Leader. State: Replica: 6e98a2dbf7a44a07a16c612b5b62ff9f, State: Running, Role: LEADER
I20260924 20:07:57.844295 2093 consensus_queue.cc:237] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } }
I20260924 20:07:57.844937 2117 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: TxnStatusTablet state changed. Reason: New leader 6e98a2dbf7a44a07a16c612b5b62ff9f. Latest consensus state: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.845036 2117 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:07:57.845276 2130 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:07:57.845459 2130 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:07:57.846095 1580 catalog_manager.cc:5495] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e98a2dbf7a44a07a16c612b5b62ff9f (127.0.115.130). New cstate: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } health_report { overall_health: UNKNOWN } } }
I20260924 20:07:57.868220 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.864184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.0.115.130:41197)
0924 20:07:57.864431 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.864435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.864448 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:07:57.864562 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.864565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.864617 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.864713 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:07:57.864997 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.865530 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.866747 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.866973 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.867049 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.867380 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:07:57.867813 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:07:57.867901 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.868036 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.868088 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260924 20:07:57.868264 2125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.864058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:41197 (local address 127.0.0.1:45514)
0924 20:07:57.864313 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.864329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.864430 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.864749 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.864752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.864760 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:07:57.864875 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.864881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.865628 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.865632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.866632 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.866643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.867076 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.867080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.867182 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.867238 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:07:57.867906 (+ 668us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:07:57.867921 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.868026 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.868122 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 20:07:57.871531 1738 raft_consensus.cc:1261] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Refusing update from remote peer 6e98a2dbf7a44a07a16c612b5b62ff9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:07:57.871616 2018 raft_consensus.cc:1261] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Refusing update from remote peer 6e98a2dbf7a44a07a16c612b5b62ff9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:07:57.872107 2117 consensus_queue.cc:1035] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:07:57.872440 2093 consensus_queue.cc:1035] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:07:57.873776 2095 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: TxnStatusTablet state changed. Reason: New leader 6e98a2dbf7a44a07a16c612b5b62ff9f. Latest consensus state: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.873890 2095 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:07:57.875017 2106 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: TxnStatusTablet state changed. Reason: New leader 6e98a2dbf7a44a07a16c612b5b62ff9f. Latest consensus state: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.875139 2106 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:07:57.875973 2093 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.876078 2093 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:07:57.876636 2095 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.876763 2095 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P 7cc79ed6cb7145639e94951912dd8cfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:07:57.876847 2117 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.876937 2117 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P 6e98a2dbf7a44a07a16c612b5b62ff9f: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:07:57.877804 2106 tablet_replica.cc:439] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 } } peers { permanent_uuid: "6e98a2dbf7a44a07a16c612b5b62ff9f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 41197 } } peers { permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 } } }
I20260924 20:07:57.877910 2106 tablet_replica.cc:442] T fad39498b2a7486a886e4612204953c4 P cd1f0aa6952348d683881f91743027b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:07:57.878298 2137 mvcc.cc:205] Tried to move back new op lower bound from 7332988837356318720 to 7332988837251641344. Current Snapshot: MvccSnapshot[applied={T|T < 7332988837356318720}]
I20260924 20:07:57.880851 2134 mvcc.cc:205] Tried to move back new op lower bound from 7332988837356318720 to 7332988837251641344. Current Snapshot: MvccSnapshot[applied={T|T < 7332988837356318720 or (T in {7332988837356318720})}]
I20260924 20:07:57.886793 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.883205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.0.115.130:41197)
0924 20:07:57.883320 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.883324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.883516 (+ 192us) server_negotiation.cc:407] Connection header received
0924 20:07:57.883640 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.883642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.883681 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.883753 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:07:57.884037 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.884520 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.885782 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.886040 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.886120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.886407 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:07:57.886473 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:07:57.886521 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.886644 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.886686 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:07:57.898089 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.894474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:37219 (local address 127.0.115.130:41197)
0924 20:07:57.894600 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.894605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.894814 (+ 209us) server_negotiation.cc:407] Connection header received
0924 20:07:57.894937 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.894939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.894979 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.895041 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:07:57.895309 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.895790 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.897055 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.897268 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.897347 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.897669 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:07:57.897735 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:07:57.897786 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.897908 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.897962 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260924 20:07:57.898157 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.894414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:41197 (local address 127.0.115.130:37219)
0924 20:07:57.894726 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:57.894741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:57.894852 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:57.895069 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:57.895072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:57.895080 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:07:57.895204 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.895210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.895882 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.895885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.896934 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:57.896946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.897377 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.897381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:57.897477 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.897533 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:07:57.897814 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:07:57.897821 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:07:57.897888 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:07:57.897970 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
I20260924 20:07:57.904470 2018 raft_consensus.cc:1261] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Refusing update from remote peer 6e98a2dbf7a44a07a16c612b5b62ff9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:07:57.904630 1738 raft_consensus.cc:1261] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Refusing update from remote peer 6e98a2dbf7a44a07a16c612b5b62ff9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:07:57.905016 2093 consensus_queue.cc:1035] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd1f0aa6952348d683881f91743027b5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:07:57.905159 2117 consensus_queue.cc:1035] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cc79ed6cb7145639e94951912dd8cfa" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:07:57.947537 1580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:07:57.947710 1580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:07:57.949879 1580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9b72457957e4349bf79eb081bc85c03: Sending DeleteTablet for 3 replicas of tablet f235db5965894da3bb0d050e978b6b2e
I20260924 20:07:57.950587 1991 tablet_service.cc:1514] Processing DeleteTablet for tablet f235db5965894da3bb0d050e978b6b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:07:57 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:35106
I20260924 20:07:57.950594 1718 tablet_service.cc:1514] Processing DeleteTablet for tablet f235db5965894da3bb0d050e978b6b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:07:57 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:35546
I20260924 20:07:57.950969 2152 tablet_replica.cc:330] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: stopping tablet replica
I20260924 20:07:57.951047 1857 tablet_service.cc:1514] Processing DeleteTablet for tablet f235db5965894da3bb0d050e978b6b2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:07:57 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:45508
I20260924 20:07:57.951263 2151 tablet_replica.cc:330] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: stopping tablet replica
I20260924 20:07:57.951351 2153 tablet_replica.cc:330] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: stopping tablet replica
I20260924 20:07:57.951473 2153 raft_consensus.cc:2229] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:07:57.951615 2152 raft_consensus.cc:2229] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:07:57.951756 2153 raft_consensus.cc:2258] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:07:57.951750 2151 raft_consensus.cc:2229] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:07:57.952219 2152 raft_consensus.cc:2258] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:07:57.952409 2153 ts_tablet_manager.cc:1905] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:07:57.952416 2151 raft_consensus.cc:2258] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:07:57.953142 2151 ts_tablet_manager.cc:1905] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:07:57.953727 2152 ts_tablet_manager.cc:1905] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:07:57.955440 2153 ts_tablet_manager.cc:1918] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:07:57.955583 2153 log.cc:1196] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/f235db5965894da3bb0d050e978b6b2e
I20260924 20:07:57.955879 2153 ts_tablet_manager.cc:1939] T f235db5965894da3bb0d050e978b6b2e P 6e98a2dbf7a44a07a16c612b5b62ff9f: Deleting consensus metadata
I20260924 20:07:57.956625 1566 catalog_manager.cc:4841] TS 6e98a2dbf7a44a07a16c612b5b62ff9f (127.0.115.130:41197): tablet f235db5965894da3bb0d050e978b6b2e (table test-table [id=6ed73099f5ee4dfa8cda1b8be5a6fd94]) successfully deleted
I20260924 20:07:57.958565 2152 ts_tablet_manager.cc:1918] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:07:57.959848 2151 ts_tablet_manager.cc:1918] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:07:57.959954 2151 log.cc:1196] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/f235db5965894da3bb0d050e978b6b2e
I20260924 20:07:57.958673 2152 log.cc:1196] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/f235db5965894da3bb0d050e978b6b2e
I20260924 20:07:57.960258 2152 ts_tablet_manager.cc:1939] T f235db5965894da3bb0d050e978b6b2e P cd1f0aa6952348d683881f91743027b5: Deleting consensus metadata
I20260924 20:07:57.960266 2151 ts_tablet_manager.cc:1939] T f235db5965894da3bb0d050e978b6b2e P 7cc79ed6cb7145639e94951912dd8cfa: Deleting consensus metadata
Sep 24 20:07:57 dist-test-slave-2pm7 krb5kdc[1536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280477, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:07:57.960965 1566 catalog_manager.cc:4841] TS cd1f0aa6952348d683881f91743027b5 (127.0.115.131:42977): tablet f235db5965894da3bb0d050e978b6b2e (table test-table [id=6ed73099f5ee4dfa8cda1b8be5a6fd94]) successfully deleted
I20260924 20:07:57.960966 1565 catalog_manager.cc:4841] TS 7cc79ed6cb7145639e94951912dd8cfa (127.0.115.129:45637): tablet f235db5965894da3bb0d050e978b6b2e (table test-table [id=6ed73099f5ee4dfa8cda1b8be5a6fd94]) successfully deleted
I20260924 20:07:57.964957 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.952261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.0.115.129:45637)
0924 20:07:57.953224 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.953234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.953629 (+ 395us) server_negotiation.cc:407] Connection header received
0924 20:07:57.953675 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.953678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.953737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.953826 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:57.954770 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.955533 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.956684 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.957021 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.957137 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.961435 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:57.961445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:57.961457 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:57.961481 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:57.963127 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.963578 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.963584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.963585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.963674 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.963978 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.963982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.963983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.964255 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:57.964465 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.964639 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.964766 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":107,"threads_started":1}
W20260924 20:07:57.965442 1778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35560
I20260924 20:07:57.974802 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:57.967156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.0.115.190:46337)
0924 20:07:57.967296 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:57.967300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:57.967443 (+ 143us) server_negotiation.cc:407] Connection header received
0924 20:07:57.967570 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:57.967573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:57.967629 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:57.967714 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:57.968430 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.968980 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.969856 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:57.970130 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:57.970252 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:57.971625 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:57.971629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:57.971632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:57.971664 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:57.973096 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.973522 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.973526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.973528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.973616 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:57.973899 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:57.973903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:57.973905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:57.974095 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:57.974262 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:57.974490 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 20:07:57.974623 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:07:57.975450 1580 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37126
I20260924 20:07:57.976832 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1653
I20260924 20:07:57.984050 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1793
I20260924 20:07:57.991283 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1933
I20260924 20:07:57.998045 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 1549
2026-09-24T20:07:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3082 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:07:58 dist-test-slave-2pm7 krb5kdc[2161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:07:58 dist-test-slave-2pm7 krb5kdc[2161](info): set up 2 sockets
Sep 24 20:07:58 dist-test-slave-2pm7 krb5kdc[2161](info): commencing operation
krb5kdc: starting...
W20260924 20:07:59.035970 462 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:07:59 dist-test-slave-2pm7 krb5kdc[2161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:07:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:07:59.156507 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:41051
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:41051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:59.271373 2174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:59.271684 2174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:59.271780 2174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:59.275799 2174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:07:59.275882 2174 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:07:59.275914 2174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:59.275971 2174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:59.276003 2174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:07:59.276041 2174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:07:59.281374 2174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:41051
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:41051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:59.282514 2174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:59.283603 2174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:59.292135 2179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:59.292240 2182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:59.292133 2180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:59.292506 2174 server_base.cc:1034] running on GCE node
I20260924 20:07:59.293150 2174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:59.294322 2174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:59.295506 2174 hybrid_clock.cc:648] HybridClock initialized: now 1790280479295479 us; error 46 us; skew 500 ppm
Sep 24 20:07:59 dist-test-slave-2pm7 krb5kdc[2161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:59.298295 2174 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:07:59.299237 2174 webserver.cc:460] Webserver started at http://127.0.115.190:41621/ using document root <none> and password file <none>
I20260924 20:07:59.299800 2174 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:59.299860 2174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:59.300120 2174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:59.302208 2174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "26d01698238d4e7a96d78ceb19f1506a"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "e37747ddd36f4470245e19d83546382e"
server_key_iv: "13b2ba015d91f9bc133345ad7d77a568"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.302779 2174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "26d01698238d4e7a96d78ceb19f1506a"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "e37747ddd36f4470245e19d83546382e"
server_key_iv: "13b2ba015d91f9bc133345ad7d77a568"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.306749 2174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:07:59.309471 2189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:59.310863 2174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:07:59.311010 2174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "26d01698238d4e7a96d78ceb19f1506a"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "e37747ddd36f4470245e19d83546382e"
server_key_iv: "13b2ba015d91f9bc133345ad7d77a568"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.311131 2174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:59.331001 2174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:59.334445 2174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:59.334656 2174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:59.343062 2174 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:41051
I20260924 20:07:59.343068 2251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:41051 every 8 connection(s)
I20260924 20:07:59.344122 2174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:07:59.347028 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:07:59.347388 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 2174
I20260924 20:07:59.347508 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:07:59.347767 462 external_mini_cluster.cc:1428] Setting key c95d6df7f9456e5a0e7433f21f6c1204
I20260924 20:07:59.353899 2252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Bootstrap starting.
I20260924 20:07:59.356302 2252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Neither blocks nor log segments found. Creating new log.
Sep 24 20:07:59 dist-test-slave-2pm7 krb5kdc[2161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:59.357125 2252 log.cc:824] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:07:59.359064 2252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: No bootstrap required, opened a new log
I20260924 20:07:59.362429 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.349235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.0.115.190:41051)
0924 20:07:59.349714 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:59.349738 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:59.349775 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:07:59.350480 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:59.350498 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:59.350842 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:59.351187 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:59.352046 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.353037 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.353919 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.354450 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.354618 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.357164 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:59.357169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:59.357184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:59.357212 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:59.359497 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.359988 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.359994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.359999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.360112 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.360402 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.360405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.360407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.360829 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:59.361037 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:59.361332 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 20:07:59.361643 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":141,"threads_started":1}
I20260924 20:07:59.362327 2252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d01698238d4e7a96d78ceb19f1506a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41051 } }
I20260924 20:07:59.362566 2252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:07:59.362627 2252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d01698238d4e7a96d78ceb19f1506a, State: Initialized, Role: FOLLOWER
I20260924 20:07:59.363037 2252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d01698238d4e7a96d78ceb19f1506a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41051 } }
I20260924 20:07:59.363157 2252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:07:59.363196 2252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:07:59.363261 2252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:07:59.364420 2252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d01698238d4e7a96d78ceb19f1506a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41051 } }
I20260924 20:07:59.364802 2252 leader_election.cc:302] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d01698238d4e7a96d78ceb19f1506a; no voters:
I20260924 20:07:59.365134 2252 leader_election.cc:288] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:07:59.365250 2257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:07:59.365494 2257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [term 1 LEADER]: Becoming Leader. State: Replica: 26d01698238d4e7a96d78ceb19f1506a, State: Running, Role: LEADER
I20260924 20:07:59.365795 2257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d01698238d4e7a96d78ceb19f1506a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41051 } }
I20260924 20:07:59.366459 2252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:07:59.367673 2258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d01698238d4e7a96d78ceb19f1506a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d01698238d4e7a96d78ceb19f1506a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41051 } } }
I20260924 20:07:59.367672 2259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d01698238d4e7a96d78ceb19f1506a. Latest consensus state: current_term: 1 leader_uuid: "26d01698238d4e7a96d78ceb19f1506a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d01698238d4e7a96d78ceb19f1506a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41051 } } }
I20260924 20:07:59.367836 2259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:59.367836 2258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a [sys.catalog]: This master's current role is: LEADER
I20260924 20:07:59.368767 2266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:07:59.371656 2266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:07:59.377036 2266 catalog_manager.cc:1347] Generated new cluster ID: cb1b66f1d4574975aabae2549ad60786
I20260924 20:07:59.377113 2266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:07:59.387696 2266 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:07:59.388705 2266 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:07:59.395480 2266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Generated new TSK 0
I20260924 20:07:59.396231 2266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:07:59.434504 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41051
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:59.562382 2278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:59.562655 2278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:59.562703 2278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:59.566747 2278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:59.566804 2278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:59.566848 2278 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:07:59.566864 2278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:07:59.566926 2278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:07:59.572609 2278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:41051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:59.573732 2278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:59.574831 2278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:59.584760 2286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:59.584754 2284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:59.584879 2278 server_base.cc:1034] running on GCE node
W20260924 20:07:59.584754 2283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:59.585462 2278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:59.586113 2278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:59.587293 2278 hybrid_clock.cc:648] HybridClock initialized: now 1790280479587264 us; error 47 us; skew 500 ppm
Sep 24 20:07:59 dist-test-slave-2pm7 krb5kdc[2161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:59.590138 2278 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:07:59.590991 2278 webserver.cc:460] Webserver started at http://127.0.115.129:39457/ using document root <none> and password file <none>
I20260924 20:07:59.591557 2278 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:59.591643 2278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:59.591879 2278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:59.593971 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "fc5c8eacb0ac4f3689406c71f319f2bb"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "c566f957a126cdfca0702a37319c83b0"
server_key_iv: "acafc9adc21f9ae6d58b56b55e86e17a"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.594547 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "fc5c8eacb0ac4f3689406c71f319f2bb"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "c566f957a126cdfca0702a37319c83b0"
server_key_iv: "acafc9adc21f9ae6d58b56b55e86e17a"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.598387 2278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:07:59.600966 2293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:59.602205 2278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:07:59.602339 2278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "fc5c8eacb0ac4f3689406c71f319f2bb"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "c566f957a126cdfca0702a37319c83b0"
server_key_iv: "acafc9adc21f9ae6d58b56b55e86e17a"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.602450 2278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:59.630584 2278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:59.634079 2278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:59.634291 2278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:59.635787 2278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:59.635857 2278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:59.635929 2278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:59.635989 2278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:07:59 dist-test-slave-2pm7 krb5kdc[2161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:59.646806 2278 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:37763
I20260924 20:07:59.647146 2408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:37763 every 8 connection(s)
I20260924 20:07:59.647702 2278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:07:59.650077 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.636837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:38645 (local address 127.0.115.190:41051)
0924 20:07:59.636995 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:59.637000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:59.637923 (+ 923us) server_negotiation.cc:407] Connection header received
0924 20:07:59.639183 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:59.639190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:59.639252 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:59.639311 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:59.640950 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.641530 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.642768 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.643225 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.643409 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.646436 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:59.646440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:59.646442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:59.646473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:59.648207 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.648761 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.648764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.648766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.648846 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.649109 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.649110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.649111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.649278 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:59.649350 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:59.649856 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 20:07:59.649958 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:07:59.650849 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.637235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41051 (local address 127.0.115.129:38645)
0924 20:07:59.637748 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:59.637801 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:59.638884 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:59.639734 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:59.639740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:59.640247 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:59.640786 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.640800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.641663 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.641667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.642597 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.642611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.643605 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.643610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.643808 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.644420 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:59.644445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:59.646253 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:59.648350 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.648356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.648370 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.648676 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.648916 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.648918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.648919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.649043 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.649365 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:59.649370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:59.649637 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 20:07:59.649799 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 20:07:59.650039 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260924 20:07:59.651818 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 2278
I20260924 20:07:59.651927 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:07:59.652277 462 external_mini_cluster.cc:1428] Setting key ef4cd37d8b0ce7d68a5a001d1bb6a99a
I20260924 20:07:59.652374 2409 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41051
I20260924 20:07:59.652546 2409 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:59.653141 2409 heartbeater.cc:511] Master 127.0.115.190:41051 requested a full tablet report, sending...
I20260924 20:07:59.654908 2205 ts_manager.cc:194] Registered new tserver with Master: fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763)
I20260924 20:07:59.656567 2205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:38645
I20260924 20:07:59.662432 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.654693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41051 (local address 127.0.115.129:59919)
0924 20:07:59.654937 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:59.654951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:59.655109 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:59.655359 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:59.655362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:59.655561 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:59.655666 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.655671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.656482 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.656485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.657133 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.657160 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.657794 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.657798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.657881 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.658380 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:59.658399 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:59.658895 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:59.660982 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.660984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.660986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.661188 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.661488 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.661491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.661492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.661569 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.662013 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:59.662015 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:59.662104 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 20:07:59.662190 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:07:59.662289 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20260924 20:07:59.662598 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.654559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:59919 (local address 127.0.115.190:41051)
0924 20:07:59.654671 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:59.654676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:59.655041 (+ 365us) server_negotiation.cc:407] Connection header received
0924 20:07:59.655214 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:59.655217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:59.655259 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:59.655343 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:59.655775 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.656355 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.657266 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.657677 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.657805 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.659063 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:59.659069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:59.659075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:59.659107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:59.660867 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.661286 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.661292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.661294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.661386 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.661659 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.661665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.661666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.661904 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:59.662013 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:59.662280 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 20:07:59.662413 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:07:59.682165 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41051
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:07:59.792800 2418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:07:59.793131 2418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:07:59.793211 2418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:07:59.797397 2418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:07:59.797508 2418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:07:59.797583 2418 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:07:59.797629 2418 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:07:59.797695 2418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:07:59.803236 2418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:41051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:07:59.804468 2418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:07:59.805598 2418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:07:59.816174 2426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:59.816279 2418 server_base.cc:1034] running on GCE node
W20260924 20:07:59.816282 2424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:07:59.816452 2423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:07:59.816989 2418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:07:59.817695 2418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:07:59.818885 2418 hybrid_clock.cc:648] HybridClock initialized: now 1790280479818864 us; error 32 us; skew 500 ppm
Sep 24 20:07:59 dist-test-slave-2pm7 krb5kdc[2161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:07:59.821746 2418 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:07:59.822624 2418 webserver.cc:460] Webserver started at http://127.0.115.130:34795/ using document root <none> and password file <none>
I20260924 20:07:59.823237 2418 fs_manager.cc:359] Metadata directory not provided
I20260924 20:07:59.823325 2418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:07:59.823550 2418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:07:59.825709 2418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "4aed5cb0f079488088bdb425f219edd6"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "c404567c4ab5c2af1183cbedf325e384"
server_key_iv: "b5fa93ef1499dca37c0a2066ab11fb51"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.826292 2418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "4aed5cb0f079488088bdb425f219edd6"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "c404567c4ab5c2af1183cbedf325e384"
server_key_iv: "b5fa93ef1499dca37c0a2066ab11fb51"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.830099 2418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:07:59.832507 2433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:59.833815 2418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:07:59.833962 2418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "4aed5cb0f079488088bdb425f219edd6"
format_stamp: "Formatted at 2026-09-24 20:07:59 on dist-test-slave-2pm7"
server_key: "c404567c4ab5c2af1183cbedf325e384"
server_key_iv: "b5fa93ef1499dca37c0a2066ab11fb51"
server_key_version: "encryptionkey@0"
I20260924 20:07:59.834090 2418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:07:59.851191 2418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:07:59.854426 2418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:07:59.854633 2418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:07:59.856148 2418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:07:59.856215 2418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:07:59.856283 2418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:07:59.856329 2418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:07:59 dist-test-slave-2pm7 krb5kdc[2161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:07:59.867125 2418 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:42355
I20260924 20:07:59.867228 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:42355 every 8 connection(s)
I20260924 20:07:59.868288 2418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:07:59.868528 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 2418
I20260924 20:07:59.868619 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:07:59.868938 462 external_mini_cluster.cc:1428] Setting key ee2e7c56609fe8853ba9e1c7d90fc9ae
I20260924 20:07:59.872630 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.857032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:59765 (local address 127.0.115.190:41051)
0924 20:07:59.857156 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:59.857160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:59.858372 (+ 1212us) server_negotiation.cc:407] Connection header received
0924 20:07:59.859459 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:59.859462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:59.859526 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:59.859636 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:59.861354 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.862253 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.864186 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.864403 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.864483 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.867671 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:59.867675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:59.867678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:59.867712 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:59.870497 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.871010 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.871015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.871017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.871106 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.871459 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.871462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.871464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.871660 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:59.871781 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:59.872347 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 20:07:59.872473 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:07:59.873461 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.857668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41051 (local address 127.0.115.130:59765)
0924 20:07:59.858228 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:59.858275 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:59.859215 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:59.859868 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:59.859879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:59.860400 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:59.861152 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.861177 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.862421 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.862426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.864050 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.864061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.864559 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.864564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.864802 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.865495 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:59.865519 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:59.867494 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:59.870614 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.870620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.870636 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.870907 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.871197 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.871201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.871203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.871356 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.871782 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:59.871790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:59.872102 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 20:07:59.872303 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 20:07:59.872636 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
I20260924 20:07:59.874805 2549 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41051
I20260924 20:07:59.874960 2549 heartbeater.cc:464] Registering TS with master...
I20260924 20:07:59.875559 2549 heartbeater.cc:511] Master 127.0.115.190:41051 requested a full tablet report, sending...
I20260924 20:07:59.876708 2205 ts_manager.cc:194] Registered new tserver with Master: 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355)
I20260924 20:07:59.877673 2205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:59765
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:07:59.885324 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.877181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:57545 (local address 127.0.115.190:41051)
0924 20:07:59.877295 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 20:07:59.877299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:07:59.877612 (+ 313us) server_negotiation.cc:407] Connection header received
0924 20:07:59.877863 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:07:59.877869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:07:59.877928 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:07:59.878022 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:07:59.878678 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.879414 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.880357 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.880564 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.880711 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.881874 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:07:59.881881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:07:59.881887 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:07:59.881919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:07:59.883427 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.884035 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.884039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.884041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.884136 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:07:59.884437 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:07:59.884440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:07:59.884441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:07:59.884660 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:07:59.884773 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:07:59.885047 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 20:07:59.885160 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 20:07:59.885337 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:07:59.877072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41051 (local address 127.0.115.130:57545)
0924 20:07:59.877488 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 20:07:59.877503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:07:59.877711 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:07:59.878170 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:07:59.878175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:07:59.878405 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:07:59.878547 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.878555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.879556 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.879559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.880241 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:07:59.880251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:07:59.880719 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:07:59.880722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:07:59.880791 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:07:59.881303 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:07:59.881323 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:07:59.881741 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:07:59.883536 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.883542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.883546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.883863 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.884246 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:07:59.884251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:07:59.884256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:07:59.884353 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:07:59.884778 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:07:59.884782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:07:59.884878 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 20:07:59.884982 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 20:07:59.885156 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:07:59.899259 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41051
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:00.014631 2558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:00.014909 2558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:00.014979 2558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:00.019090 2558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:00.019150 2558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:00.019191 2558 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:08:00.019207 2558 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:08:00.019244 2558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:00.024611 2558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:41051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:00.025648 2558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:00.026614 2558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:00.036327 2558 server_base.cc:1034] running on GCE node
W20260924 20:08:00.036355 2564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:00.036347 2563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:00.036309 2566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:00.036963 2558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:00.037643 2558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:00.038796 2558 hybrid_clock.cc:648] HybridClock initialized: now 1790280480038765 us; error 44 us; skew 500 ppm
Sep 24 20:08:00 dist-test-slave-2pm7 krb5kdc[2161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:00.041543 2558 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:00.042343 2558 webserver.cc:460] Webserver started at http://127.0.115.131:37145/ using document root <none> and password file <none>
I20260924 20:08:00.042865 2558 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:00.042924 2558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:00.043088 2558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:00.045174 2558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "b8f45831ae88444e959f0b7cb6803850"
format_stamp: "Formatted at 2026-09-24 20:08:00 on dist-test-slave-2pm7"
server_key: "0bab898639889f3a04bdb8c16cc0e4b2"
server_key_iv: "cf4650f6ff780e64053b65b36a8b1597"
server_key_version: "encryptionkey@0"
I20260924 20:08:00.045714 2558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "b8f45831ae88444e959f0b7cb6803850"
format_stamp: "Formatted at 2026-09-24 20:08:00 on dist-test-slave-2pm7"
server_key: "0bab898639889f3a04bdb8c16cc0e4b2"
server_key_iv: "cf4650f6ff780e64053b65b36a8b1597"
server_key_version: "encryptionkey@0"
I20260924 20:08:00.049408 2558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:08:00.051798 2573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:00.053136 2558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:00.053264 2558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "b8f45831ae88444e959f0b7cb6803850"
format_stamp: "Formatted at 2026-09-24 20:08:00 on dist-test-slave-2pm7"
server_key: "0bab898639889f3a04bdb8c16cc0e4b2"
server_key_iv: "cf4650f6ff780e64053b65b36a8b1597"
server_key_version: "encryptionkey@0"
I20260924 20:08:00.053363 2558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:00.073966 2558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:00.077431 2558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:00.077636 2558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:00.079238 2558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:00.079310 2558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:00.079382 2558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:00.079429 2558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:08:00 dist-test-slave-2pm7 krb5kdc[2161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:00.090497 2558 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:42343
I20260924 20:08:00.090569 2688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:42343 every 8 connection(s)
I20260924 20:08:00.091444 2558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:00.093487 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.080176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:59309 (local address 127.0.115.190:41051)
0924 20:08:00.080330 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.080335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.081312 (+ 977us) server_negotiation.cc:407] Connection header received
0924 20:08:00.082607 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.082611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.082681 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.082781 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:00.084352 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.084903 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.086306 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.086531 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.086645 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.089650 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:00.089654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:00.089656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:00.089683 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:00.091349 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.092004 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.092008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.092010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.092113 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.092469 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.092472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.092473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.092712 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:00.092797 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.093255 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.093368 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:08:00.094254 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.080608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41051 (local address 127.0.115.131:59309)
0924 20:08:00.081123 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.081186 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.082355 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.082972 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.082982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.083456 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:00.084189 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.084214 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.085033 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.085037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.086183 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.086193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.086698 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.086702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.086883 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.087497 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:00.087522 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:00.089444 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:00.091464 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:00.091470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:00.091486 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:00.091843 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:00.092232 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:00.092235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:00.092237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:00.092378 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:00.092806 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:00.092810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:00.093030 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.093173 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.093414 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
I20260924 20:08:00.095449 2689 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41051
I20260924 20:08:00.095605 2689 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:00.095909 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 2558
I20260924 20:08:00.096050 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:00.096194 2689 heartbeater.cc:511] Master 127.0.115.190:41051 requested a full tablet report, sending...
I20260924 20:08:00.096422 462 external_mini_cluster.cc:1428] Setting key 2181a3ac13a2b5102e9792eb46eace98
I20260924 20:08:00.097337 2205 ts_manager.cc:194] Registered new tserver with Master: b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:00.098132 2205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:59309
I20260924 20:08:00.098645 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:08:00.106065 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.097761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41051 (local address 127.0.115.131:46631)
0924 20:08:00.098022 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.098037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.098121 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.098353 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.098355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.098567 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:00.098748 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.098758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.099796 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.099802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.100827 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.100838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.101452 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.101454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.101521 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.101999 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:00.102010 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:00.102308 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:00.104377 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:00.104381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:00.104383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:00.104647 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:00.104918 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:00.104924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:00.104929 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:00.105035 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:00.105393 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:00.105400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:00.105515 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.105639 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.105820 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260924 20:08:00.106065 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.097718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:46631 (local address 127.0.115.190:41051)
0924 20:08:00.098096 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.098101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.098127 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:08:00.098210 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.098213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.098251 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.098331 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:00.098875 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.099624 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.100950 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.101341 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.101462 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.102447 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:00.102453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:00.102459 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:00.102490 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:00.104274 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.104746 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.104748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.104750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.104823 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.105147 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.105149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.105150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.105303 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:00.105375 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.105723 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.105855 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
Sep 24 20:08:00 dist-test-slave-2pm7 krb5kdc[2161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:08:00 dist-test-slave-2pm7 krb5kdc[2161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:00.125574 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.117664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.0.115.190:41051)
0924 20:08:00.117780 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.117784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.117795 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:08:00.117831 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.117834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.117867 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.117932 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:00.118598 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.119152 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.120084 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.120312 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.120416 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.122789 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:00.122792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:00.122794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:00.122817 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:00.124221 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.124613 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.124616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.124617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.124688 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.124911 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.124913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.124914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.125079 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:00.125177 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.125366 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.125457 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:08:00.128537 2205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:00.132160 2205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:00.147202 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.141843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.0.115.129:37763)
0924 20:08:00.142970 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.142983 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.143001 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:08:00.143062 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.143066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.143208 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.143550 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:00.143696 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.144505 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.145945 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.146480 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.146599 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.146784 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.146952 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.147041 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":89,"threads_started":1}
I20260924 20:08:00.147195 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.141481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:37763 (local address 127.0.0.1:56472)
0924 20:08:00.142196 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.142249 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.142398 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.143352 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.143354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.143371 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:00.143580 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.143590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.144617 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.144623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.145819 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.145829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.146600 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.146604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.146722 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.146765 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.146914 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.147041 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":48,"spinlock_wait_cycles":10496,"thread_start_us":103,"threads_started":1}
I20260924 20:08:00.148380 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.141481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.0.115.131:42343)
0924 20:08:00.142816 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.142826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.142847 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:08:00.142915 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.142919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.143090 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.143262 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:00.143632 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.144997 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.146294 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.147149 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.147864 (+ 715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.148054 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.148115 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.148226 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":121,"threads_started":1}
I20260924 20:08:00.148883 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.141292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:42343 (local address 127.0.0.1:33382)
0924 20:08:00.142195 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.142258 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.142400 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.143250 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.143256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.143281 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:00.143485 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.143493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.145166 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.145170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.146164 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.146174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.147653 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.147657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.147762 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.147788 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.147853 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.147924 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":40,"spinlock_wait_cycles":2944,"thread_start_us":92,"threads_started":1}
I20260924 20:08:00.149251 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.142047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:42355 (local address 127.0.0.1:47522)
0924 20:08:00.142643 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.142655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.142750 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.143428 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.143434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.143454 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:00.143604 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.143614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.145166 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.145170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.147479 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.147490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.148765 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.148769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.148968 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.148992 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.149064 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.149125 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":70,"thread_start_us":89,"threads_started":1}
I20260924 20:08:00.149753 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.142437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.0.115.130:42355)
0924 20:08:00.142849 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.142861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.142883 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:08:00.142956 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.142961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.143159 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.143324 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:00.143744 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.145035 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.147634 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.148443 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.148545 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.148690 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.149512 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.149610 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":95,"threads_started":1}
I20260924 20:08:00.150642 2343 tablet_service.cc:1467] Processing CreateTablet for tablet a96a7d77080b44e7a7902b1e0a522c44 (DEFAULT_TABLE table=test-table [id=d534c436b7ec4c239a100ca8584dde96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:00.151628 2343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a7d77080b44e7a7902b1e0a522c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.151641 2623 tablet_service.cc:1467] Processing CreateTablet for tablet a96a7d77080b44e7a7902b1e0a522c44 (DEFAULT_TABLE table=test-table [id=d534c436b7ec4c239a100ca8584dde96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:00.152798 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a7d77080b44e7a7902b1e0a522c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.152335 2482 tablet_service.cc:1467] Processing CreateTablet for tablet a96a7d77080b44e7a7902b1e0a522c44 (DEFAULT_TABLE table=test-table [id=d534c436b7ec4c239a100ca8584dde96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:00.153406 2482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a7d77080b44e7a7902b1e0a522c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.157914 2712 tablet_bootstrap.cc:492] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Bootstrap starting.
I20260924 20:08:00.160704 2712 tablet_bootstrap.cc:654] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.161649 2712 log.cc:824] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:00.161746 2713 tablet_bootstrap.cc:492] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Bootstrap starting.
I20260924 20:08:00.161746 2714 tablet_bootstrap.cc:492] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Bootstrap starting.
I20260924 20:08:00.163520 2712 tablet_bootstrap.cc:492] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: No bootstrap required, opened a new log
I20260924 20:08:00.163728 2712 ts_tablet_manager.cc:1397] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:08:00.163998 2713 tablet_bootstrap.cc:654] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.163998 2714 tablet_bootstrap.cc:654] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.164848 2713 log.cc:824] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:00.164848 2714 log.cc:824] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:00.166761 2713 tablet_bootstrap.cc:492] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: No bootstrap required, opened a new log
I20260924 20:08:00.166844 2714 tablet_bootstrap.cc:492] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: No bootstrap required, opened a new log
I20260924 20:08:00.166958 2713 ts_tablet_manager.cc:1397] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:08:00.167040 2714 ts_tablet_manager.cc:1397] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:08:00.167308 2712 raft_consensus.cc:348] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.167564 2712 raft_consensus.cc:374] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.167613 2712 raft_consensus.cc:729] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c8eacb0ac4f3689406c71f319f2bb, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.168087 2712 consensus_queue.cc:260] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.168767 2409 heartbeater.cc:503] Master 127.0.115.190:41051 was elected leader, sending a full tablet report...
I20260924 20:08:00.169070 2712 ts_tablet_manager.cc:1428] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:08:00.170626 2713 raft_consensus.cc:348] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.170816 2713 raft_consensus.cc:374] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.170848 2713 raft_consensus.cc:729] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.170650 2714 raft_consensus.cc:348] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.170903 2714 raft_consensus.cc:374] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.170951 2714 raft_consensus.cc:729] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f45831ae88444e959f0b7cb6803850, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.171267 2713 consensus_queue.cc:260] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.171368 2714 consensus_queue.cc:260] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.171878 2549 heartbeater.cc:503] Master 127.0.115.190:41051 was elected leader, sending a full tablet report...
I20260924 20:08:00.172325 2713 ts_tablet_manager.cc:1428] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:08:00.172525 2714 ts_tablet_manager.cc:1428] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:08:00.172739 2689 heartbeater.cc:503] Master 127.0.115.190:41051 was elected leader, sending a full tablet report...
W20260924 20:08:00.342474 2690 tablet.cc:2369] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:08:00.369740 2550 tablet.cc:2369] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:00.369731 2718 raft_consensus.cc:482] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:00.369968 2718 raft_consensus.cc:504] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.371182 2718 leader_election.cc:288] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343), 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355)
I20260924 20:08:00.376530 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.372011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:55715 (local address 127.0.115.130:42355)
0924 20:08:00.372150 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.372156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.372327 (+ 171us) server_negotiation.cc:407] Connection header received
0924 20:08:00.372431 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.372433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.372469 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.372557 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:00.372778 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.373498 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.375127 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.375773 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.375894 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.375982 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.376318 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.376395 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:08:00.376530 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.371588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:49559 (local address 127.0.115.131:42343)
0924 20:08:00.371717 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.371721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.371824 (+ 103us) server_negotiation.cc:407] Connection header received
0924 20:08:00.372057 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.372063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.372122 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.372220 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:00.372555 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.373378 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.375127 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.375773 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.375894 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.375982 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.376318 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.376395 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:08:00.376631 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.371873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:42355 (local address 127.0.115.129:55715)
0924 20:08:00.372229 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.372243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.372352 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.372580 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.372582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.372598 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:00.372683 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.372687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.373601 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.373604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.374987 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.374994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.375894 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.375898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.376128 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.376190 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.376275 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.376415 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":161,"spinlock_wait_cycles":2688,"thread_start_us":139,"threads_started":1}
I20260924 20:08:00.376631 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.371456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:42343 (local address 127.0.115.129:49559)
0924 20:08:00.371710 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.371727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.371862 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.372257 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.372263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.372288 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:00.372445 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.372452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.373490 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.373493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.374981 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.374994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.375894 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.375898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.376132 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.376190 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.376276 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.376415 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":185,"spinlock_wait_cycles":14720,"thread_start_us":98,"threads_started":1}
I20260924 20:08:00.377422 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a7d77080b44e7a7902b1e0a522c44" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850" is_pre_election: true
I20260924 20:08:00.377758 2643 raft_consensus.cc:2454] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 0.
I20260924 20:08:00.378244 2503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a7d77080b44e7a7902b1e0a522c44" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aed5cb0f079488088bdb425f219edd6" is_pre_election: true
I20260924 20:08:00.378475 2503 raft_consensus.cc:2454] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 0.
I20260924 20:08:00.378976 2296 leader_election.cc:302] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f45831ae88444e959f0b7cb6803850, fc5c8eacb0ac4f3689406c71f319f2bb; no voters:
I20260924 20:08:00.379288 2718 raft_consensus.cc:2781] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:00.379410 2718 raft_consensus.cc:482] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:00.379485 2718 raft_consensus.cc:3037] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.380599 2718 raft_consensus.cc:504] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.380954 2718 leader_election.cc:288] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 election: Requested vote from peers b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343), 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355)
I20260924 20:08:00.381384 2503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a7d77080b44e7a7902b1e0a522c44" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aed5cb0f079488088bdb425f219edd6"
I20260924 20:08:00.381403 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96a7d77080b44e7a7902b1e0a522c44" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850"
I20260924 20:08:00.381518 2503 raft_consensus.cc:3037] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.381518 2643 raft_consensus.cc:3037] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.382934 2503 raft_consensus.cc:2454] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 1.
I20260924 20:08:00.383248 2296 leader_election.cc:302] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, fc5c8eacb0ac4f3689406c71f319f2bb; no voters:
I20260924 20:08:00.383343 2643 raft_consensus.cc:2454] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 1.
I20260924 20:08:00.383440 2718 raft_consensus.cc:2781] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:00.383685 2718 raft_consensus.cc:686] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 LEADER]: Becoming Leader. State: Replica: fc5c8eacb0ac4f3689406c71f319f2bb, State: Running, Role: LEADER
I20260924 20:08:00.384270 2718 consensus_queue.cc:237] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } }
I20260924 20:08:00.387367 2206 catalog_manager.cc:5495] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb reported cstate change: term changed from 0 to 1, leader changed from <none> to fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129). New cstate: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } health_report { overall_health: UNKNOWN } } }
W20260924 20:08:00.399240 2410 tablet.cc:2369] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 20:08:00 dist-test-slave-2pm7 krb5kdc[2161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:00.466615 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.458014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.0.115.190:41051)
0924 20:08:00.458217 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.458234 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.458466 (+ 232us) server_negotiation.cc:407] Connection header received
0924 20:08:00.458688 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.458695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.458732 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.458814 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:00.459477 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.460051 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.460861 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.461068 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.461174 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.463449 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:00.463453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:00.463455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:00.463475 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:00.464803 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.465308 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.465313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.465315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.465393 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:00.465656 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:00.465661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:00.465663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:00.465849 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:00.465951 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.466282 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.466466 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":43}
I20260924 20:08:00.466637 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.457985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41051 (local address 127.0.0.1:57214)
0924 20:08:00.458362 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.458387 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.458527 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.458834 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.458836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.459264 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:00.459357 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.459362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.460167 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.460171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.460738 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.460746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.461180 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.461183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.461255 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.461848 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:00.461860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:00.463307 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:00.464941 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:00.464952 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:00.464960 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:00.465204 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:00.465487 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:00.465491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:00.465492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:00.465571 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:00.465982 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:00.465990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:00.466101 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.466233 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.466409 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
I20260924 20:08:00.470856 2206 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:57214:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:08:00.471545 2206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:00.476852 2343 tablet_service.cc:1467] Processing CreateTablet for tablet 26c267efb1b44d728343d0496cb896e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b8651be3fdd41e6b7e7bcce1a93e359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:08:00.476840 2623 tablet_service.cc:1467] Processing CreateTablet for tablet 26c267efb1b44d728343d0496cb896e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b8651be3fdd41e6b7e7bcce1a93e359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:08:00.476836 2482 tablet_service.cc:1467] Processing CreateTablet for tablet 26c267efb1b44d728343d0496cb896e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b8651be3fdd41e6b7e7bcce1a93e359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:08:00.477648 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c267efb1b44d728343d0496cb896e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.477648 2482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c267efb1b44d728343d0496cb896e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.477653 2343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c267efb1b44d728343d0496cb896e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.480640 2712 tablet_bootstrap.cc:492] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: Bootstrap starting.
I20260924 20:08:00.480756 2713 tablet_bootstrap.cc:492] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: Bootstrap starting.
I20260924 20:08:00.481341 2714 tablet_bootstrap.cc:492] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: Bootstrap starting.
I20260924 20:08:00.481911 2713 tablet_bootstrap.cc:654] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.482056 2712 tablet_bootstrap.cc:654] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.482415 2714 tablet_bootstrap.cc:654] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.483378 2713 tablet_bootstrap.cc:492] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: No bootstrap required, opened a new log
I20260924 20:08:00.483489 2713 ts_tablet_manager.cc:1397] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:00.483843 2712 tablet_bootstrap.cc:492] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: No bootstrap required, opened a new log
I20260924 20:08:00.483843 2714 tablet_bootstrap.cc:492] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: No bootstrap required, opened a new log
I20260924 20:08:00.483968 2714 ts_tablet_manager.cc:1397] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:00.483974 2712 ts_tablet_manager.cc:1397] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:00.483928 2713 raft_consensus.cc:348] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.484078 2713 raft_consensus.cc:374] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.484124 2713 raft_consensus.cc:729] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.484275 2713 consensus_queue.cc:260] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.484444 2712 raft_consensus.cc:348] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.484444 2714 raft_consensus.cc:348] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.484534 2713 ts_tablet_manager.cc:1428] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:08:00.484565 2714 raft_consensus.cc:374] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.484565 2712 raft_consensus.cc:374] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.484601 2712 raft_consensus.cc:729] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c8eacb0ac4f3689406c71f319f2bb, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.484601 2714 raft_consensus.cc:729] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f45831ae88444e959f0b7cb6803850, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.484732 2714 consensus_queue.cc:260] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.484732 2712 consensus_queue.cc:260] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.484985 2712 ts_tablet_manager.cc:1428] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:08:00.484984 2714 ts_tablet_manager.cc:1428] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:00.485564 2719 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.485735 2719 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:00.485948 2720 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.486090 2720 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:00.486271 2718 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.486397 2718 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:00.590339 2718 raft_consensus.cc:482] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:00.590528 2718 raft_consensus.cc:504] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.590998 2718 leader_election.cc:288] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:00.591580 2503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c267efb1b44d728343d0496cb896e4" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aed5cb0f079488088bdb425f219edd6" is_pre_election: true
I20260924 20:08:00.591619 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c267efb1b44d728343d0496cb896e4" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850" is_pre_election: true
I20260924 20:08:00.591737 2503 raft_consensus.cc:2454] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 0.
I20260924 20:08:00.591813 2643 raft_consensus.cc:2454] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 0.
I20260924 20:08:00.592115 2296 leader_election.cc:302] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, fc5c8eacb0ac4f3689406c71f319f2bb; no voters:
I20260924 20:08:00.592285 2718 raft_consensus.cc:2781] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:00.592346 2718 raft_consensus.cc:482] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:00.592384 2718 raft_consensus.cc:3037] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.593411 2718 raft_consensus.cc:504] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.593753 2718 leader_election.cc:288] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 election: Requested vote from peers 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:00.594097 2503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c267efb1b44d728343d0496cb896e4" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aed5cb0f079488088bdb425f219edd6"
I20260924 20:08:00.594211 2503 raft_consensus.cc:3037] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.594170 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c267efb1b44d728343d0496cb896e4" candidate_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850"
I20260924 20:08:00.594309 2643 raft_consensus.cc:3037] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.595278 2643 raft_consensus.cc:2454] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 1.
I20260924 20:08:00.595348 2503 raft_consensus.cc:2454] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5c8eacb0ac4f3689406c71f319f2bb in term 1.
I20260924 20:08:00.595613 2296 leader_election.cc:302] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f45831ae88444e959f0b7cb6803850, fc5c8eacb0ac4f3689406c71f319f2bb; no voters:
I20260924 20:08:00.595813 2718 raft_consensus.cc:2781] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:00.595927 2718 raft_consensus.cc:686] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 LEADER]: Becoming Leader. State: Replica: fc5c8eacb0ac4f3689406c71f319f2bb, State: Running, Role: LEADER
I20260924 20:08:00.596110 2718 consensus_queue.cc:237] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.596848 2723 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: TxnStatusTablet state changed. Reason: New leader fc5c8eacb0ac4f3689406c71f319f2bb. Latest consensus state: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.596967 2723 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:08:00.597421 2735 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:08:00.597673 2735 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:08:00.598073 2200 catalog_manager.cc:5495] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb reported cstate change: term changed from 0 to 1, leader changed from <none> to fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129). New cstate: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:00.637786 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.633884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:37763 (local address 127.0.0.1:60266)
0924 20:08:00.634090 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.634108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.634236 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.634586 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.634589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.634597 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:00.634712 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.634717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.635435 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.635439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.636114 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.636125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.636565 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.636568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.636674 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.636720 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:00.637468 (+ 748us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:00.637485 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.637556 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.637645 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 20:08:00.637786 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.634015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.0.115.129:37763)
0924 20:08:00.634156 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.634161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.634176 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:00.634402 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.634404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.634464 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.634571 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:00.634820 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.635324 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.636238 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.636457 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.636562 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.636871 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:00.637374 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:00.637499 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.637565 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.637626 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:08:00.643569 2503 raft_consensus.cc:1261] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Refusing update from remote peer fc5c8eacb0ac4f3689406c71f319f2bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:00.643569 2643 raft_consensus.cc:1261] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Refusing update from remote peer fc5c8eacb0ac4f3689406c71f319f2bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:00.644344 2718 consensus_queue.cc:1035] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:00.644569 2723 consensus_queue.cc:1035] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:00.648221 2720 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: TxnStatusTablet state changed. Reason: New leader fc5c8eacb0ac4f3689406c71f319f2bb. Latest consensus state: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.648348 2720 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:00.650738 2719 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: TxnStatusTablet state changed. Reason: New leader fc5c8eacb0ac4f3689406c71f319f2bb. Latest consensus state: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.650862 2719 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:00.651793 2718 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.651906 2718 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:08:00.652565 2720 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.652678 2720 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P b8f45831ae88444e959f0b7cb6803850: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:00.657920 2718 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.658032 2718 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P fc5c8eacb0ac4f3689406c71f319f2bb: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:08:00.660298 2719 tablet_replica.cc:439] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } } }
I20260924 20:08:00.660409 2719 tablet_replica.cc:442] T 26c267efb1b44d728343d0496cb896e4 P 4aed5cb0f079488088bdb425f219edd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:00.662595 2738 mvcc.cc:205] Tried to move back new op lower bound from 7332988848709029888 to 7332988848523001856. Current Snapshot: MvccSnapshot[applied={T|T < 7332988848709029888 or (T in {7332988848709029888})}]
I20260924 20:08:00.664637 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.658056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.0.115.129:37763)
0924 20:08:00.658215 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.658219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.660647 (+ 2428us) server_negotiation.cc:407] Connection header received
0924 20:08:00.660854 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.660858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.660905 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.660948 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:00.661306 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.662085 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.663380 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.663715 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.663791 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.664174 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:00.664254 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:00.664300 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.664462 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.664519 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 20:08:00.668449 2740 mvcc.cc:205] Tried to move back new op lower bound from 7332988848709029888 to 7332988848523001856. Current Snapshot: MvccSnapshot[applied={T|T < 7332988848709029888}]
I20260924 20:08:00.676750 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.672757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:46903 (local address 127.0.115.129:37763)
0924 20:08:00.672879 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.672884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.673095 (+ 211us) server_negotiation.cc:407] Connection header received
0924 20:08:00.673225 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.673227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.673269 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.673328 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:00.673607 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.674133 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.675392 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.675712 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.675823 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.676260 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:00.676367 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:00.676435 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.676548 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.676597 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:08:00.677122 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.672690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:37763 (local address 127.0.115.129:46903)
0924 20:08:00.673008 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.673024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.673139 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.673357 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.673360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.673379 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:00.673503 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.673518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.674231 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.674234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.675270 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.675281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.675901 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.675904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.676024 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.676075 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:00.676466 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:00.676470 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.676903 (+ 433us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.676988 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":114,"threads_started":1}
I20260924 20:08:00.683776 2643 raft_consensus.cc:1261] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Refusing update from remote peer fc5c8eacb0ac4f3689406c71f319f2bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:00.683820 2503 raft_consensus.cc:1261] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Refusing update from remote peer fc5c8eacb0ac4f3689406c71f319f2bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:00.684229 2746 consensus_queue.cc:1035] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:00.684405 2718 consensus_queue.cc:1035] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:00.691088 2739 mvcc.cc:205] Tried to move back new op lower bound from 7332988848877645824 to 7332988847656558592. Current Snapshot: MvccSnapshot[applied={T|T < 7332988848877645824}]
I20260924 20:08:00.728046 2200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:00.728291 2200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:00.730929 2200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Sending DeleteTablet for 3 replicas of tablet a96a7d77080b44e7a7902b1e0a522c44
I20260924 20:08:00.731690 2343 tablet_service.cc:1514] Processing DeleteTablet for tablet a96a7d77080b44e7a7902b1e0a522c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:00 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:56472
I20260924 20:08:00.731727 2482 tablet_service.cc:1514] Processing DeleteTablet for tablet a96a7d77080b44e7a7902b1e0a522c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:00 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:47522
I20260924 20:08:00.732246 2755 tablet_replica.cc:330] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: stopping tablet replica
I20260924 20:08:00.732409 2200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:08:00.732568 2755 raft_consensus.cc:2229] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:00.732652 2623 tablet_service.cc:1514] Processing DeleteTablet for tablet a96a7d77080b44e7a7902b1e0a522c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:00 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:33382
W20260924 20:08:00.732813 2200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:00.732944 2755 raft_consensus.cc:2258] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:00.733286 2757 tablet_replica.cc:330] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: stopping tablet replica
I20260924 20:08:00.733436 2756 tablet_replica.cc:330] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: stopping tablet replica
I20260924 20:08:00.733626 2757 raft_consensus.cc:2229] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:00.733827 2756 raft_consensus.cc:2229] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:00.733871 2757 raft_consensus.cc:2258] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:00.734114 2756 raft_consensus.cc:2258] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:00.735507 2757 ts_tablet_manager.cc:1905] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:00.735509 2756 ts_tablet_manager.cc:1905] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:00.738483 2343 tablet_service.cc:1467] Processing CreateTablet for tablet da08319889a44a3e80d4e8f9c5bcdba5 (DEFAULT_TABLE table=test-table [id=6b85541126d44ddaaae472f2f82f5884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:00.738734 2482 tablet_service.cc:1467] Processing CreateTablet for tablet da08319889a44a3e80d4e8f9c5bcdba5 (DEFAULT_TABLE table=test-table [id=6b85541126d44ddaaae472f2f82f5884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:00.738804 2343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da08319889a44a3e80d4e8f9c5bcdba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.738976 2482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da08319889a44a3e80d4e8f9c5bcdba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.740577 2755 ts_tablet_manager.cc:1905] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:00.741392 2712 tablet_bootstrap.cc:492] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: Bootstrap starting.
I20260924 20:08:00.741562 2623 tablet_service.cc:1467] Processing CreateTablet for tablet da08319889a44a3e80d4e8f9c5bcdba5 (DEFAULT_TABLE table=test-table [id=6b85541126d44ddaaae472f2f82f5884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:00.741818 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da08319889a44a3e80d4e8f9c5bcdba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:00.742375 2712 tablet_bootstrap.cc:654] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.744012 2757 ts_tablet_manager.cc:1918] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:00.744112 2757 log.cc:1196] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/a96a7d77080b44e7a7902b1e0a522c44
I20260924 20:08:00.744421 2757 ts_tablet_manager.cc:1939] T a96a7d77080b44e7a7902b1e0a522c44 P b8f45831ae88444e959f0b7cb6803850: Deleting consensus metadata
I20260924 20:08:00.744489 2713 tablet_bootstrap.cc:492] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: Bootstrap starting.
I20260924 20:08:00.745091 2755 ts_tablet_manager.cc:1918] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:00.745177 2755 log.cc:1196] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/a96a7d77080b44e7a7902b1e0a522c44
I20260924 20:08:00.745446 2713 tablet_bootstrap.cc:654] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.745515 2755 ts_tablet_manager.cc:1939] T a96a7d77080b44e7a7902b1e0a522c44 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting consensus metadata
I20260924 20:08:00.746200 2714 tablet_bootstrap.cc:492] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: Bootstrap starting.
I20260924 20:08:00.746343 2712 tablet_bootstrap.cc:492] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: No bootstrap required, opened a new log
I20260924 20:08:00.746420 2712 ts_tablet_manager.cc:1397] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 20:08:00.746960 2712 raft_consensus.cc:348] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.747071 2712 raft_consensus.cc:374] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.747103 2712 raft_consensus.cc:729] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c8eacb0ac4f3689406c71f319f2bb, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.747169 2714 tablet_bootstrap.cc:654] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:00.747247 2712 consensus_queue.cc:260] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.747540 2712 ts_tablet_manager.cc:1428] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:00.747884 2192 catalog_manager.cc:4841] TS b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343): tablet a96a7d77080b44e7a7902b1e0a522c44 (table test-table [id=d534c436b7ec4c239a100ca8584dde96]) successfully deleted
I20260924 20:08:00.748363 2714 tablet_bootstrap.cc:492] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: No bootstrap required, opened a new log
I20260924 20:08:00.748459 2714 ts_tablet_manager.cc:1397] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:00.748588 2756 ts_tablet_manager.cc:1918] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:00.748750 2756 log.cc:1196] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/a96a7d77080b44e7a7902b1e0a522c44
I20260924 20:08:00.748930 2714 raft_consensus.cc:348] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.749058 2714 raft_consensus.cc:374] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.749116 2714 raft_consensus.cc:729] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f45831ae88444e959f0b7cb6803850, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.749248 2714 consensus_queue.cc:260] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.749310 2756 ts_tablet_manager.cc:1939] T a96a7d77080b44e7a7902b1e0a522c44 P 4aed5cb0f079488088bdb425f219edd6: Deleting consensus metadata
I20260924 20:08:00.749522 2714 ts_tablet_manager.cc:1428] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:00.750782 2192 catalog_manager.cc:4841] TS 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355): tablet a96a7d77080b44e7a7902b1e0a522c44 (table test-table [id=d534c436b7ec4c239a100ca8584dde96]) successfully deleted
I20260924 20:08:00.751500 2713 tablet_bootstrap.cc:492] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: No bootstrap required, opened a new log
I20260924 20:08:00.751583 2713 ts_tablet_manager.cc:1397] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260924 20:08:00.751750 2193 catalog_manager.cc:4841] TS fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763): tablet a96a7d77080b44e7a7902b1e0a522c44 (table test-table [id=d534c436b7ec4c239a100ca8584dde96]) successfully deleted
I20260924 20:08:00.752055 2713 raft_consensus.cc:348] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.752192 2713 raft_consensus.cc:374] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:00.752230 2713 raft_consensus.cc:729] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Initialized, Role: FOLLOWER
I20260924 20:08:00.752377 2713 consensus_queue.cc:260] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.752657 2713 ts_tablet_manager.cc:1428] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:00.987309 2719 raft_consensus.cc:482] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:00.987466 2719 raft_consensus.cc:504] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.988740 2719 leader_election.cc:288] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:00.992883 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.988943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:37763 (local address 127.0.115.130:45957)
0924 20:08:00.989290 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.989306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.989436 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.989813 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.989816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.989832 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:00.989967 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.989974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.990732 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.990736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.991665 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.991675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.992398 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.992402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.992512 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.992539 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.992622 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.992707 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":53,"thread_start_us":96,"threads_started":1}
I20260924 20:08:00.992897 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.989062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:45957 (local address 127.0.115.129:37763)
0924 20:08:00.989205 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.989210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.989397 (+ 187us) server_negotiation.cc:407] Connection header received
0924 20:08:00.989551 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.989555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.989613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.989694 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:00.990082 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.990622 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.991814 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.992278 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.992373 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.992420 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.992650 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.992768 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:08:00.992983 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.989128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:37725 (local address 127.0.115.131:42343)
0924 20:08:00.989365 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:00.989382 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:00.989554 (+ 172us) server_negotiation.cc:407] Connection header received
0924 20:08:00.989800 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:00.989803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:00.989854 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:00.989942 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:00.990191 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.990739 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.991773 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.992409 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.992533 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.992657 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:00.992764 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 20:08:00.992837 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260924 20:08:00.993355 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:00.989046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:42343 (local address 127.0.115.130:37725)
0924 20:08:00.989437 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:00.989454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:00.989570 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:00.989962 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:00.989964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:00.989979 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:00.990078 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.990084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.990855 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.990858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.991649 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:00.991658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:00.992534 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:00.992537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:00.992627 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:00.992650 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:08:00.992729 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:08:00.992802 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260924 20:08:00.993454 2363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da08319889a44a3e80d4e8f9c5bcdba5" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" is_pre_election: true
I20260924 20:08:00.993757 2363 raft_consensus.cc:2454] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 0.
I20260924 20:08:00.993916 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da08319889a44a3e80d4e8f9c5bcdba5" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850" is_pre_election: true
I20260924 20:08:00.994081 2643 raft_consensus.cc:2454] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 0.
I20260924 20:08:00.994246 2437 leader_election.cc:302] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, fc5c8eacb0ac4f3689406c71f319f2bb; no voters:
I20260924 20:08:00.994549 2719 raft_consensus.cc:2781] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:00.994620 2719 raft_consensus.cc:482] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:00.994659 2719 raft_consensus.cc:3037] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.995687 2719 raft_consensus.cc:504] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:00.996109 2719 leader_election.cc:288] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 election: Requested vote from peers fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:00.996438 2363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da08319889a44a3e80d4e8f9c5bcdba5" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb"
I20260924 20:08:00.996438 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da08319889a44a3e80d4e8f9c5bcdba5" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850"
I20260924 20:08:00.996559 2363 raft_consensus.cc:3037] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.996567 2643 raft_consensus.cc:3037] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:00.997687 2643 raft_consensus.cc:2454] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 1.
I20260924 20:08:00.997687 2363 raft_consensus.cc:2454] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 1.
I20260924 20:08:00.998345 2436 leader_election.cc:302] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, b8f45831ae88444e959f0b7cb6803850; no voters:
I20260924 20:08:00.998538 2719 raft_consensus.cc:2781] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:00.998765 2719 raft_consensus.cc:686] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 1 LEADER]: Becoming Leader. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Running, Role: LEADER
I20260924 20:08:00.999013 2719 consensus_queue.cc:237] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:01.001467 2205 catalog_manager.cc:5495] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130). New cstate: current_term: 1 leader_uuid: "4aed5cb0f079488088bdb425f219edd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:01.046561 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:01.042346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.0.115.130:42355)
0924 20:08:01.042602 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:01.042608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:01.042624 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:01.042800 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:01.042803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:01.042858 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:01.042965 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:01.043215 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.043779 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.044809 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.045076 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.045325 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:01.045631 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:01.046119 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:01.046254 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:01.046316 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 20:08:01.046377 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":110,"threads_started":1}
I20260924 20:08:01.059087 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:01.054923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:37763 (local address 127.0.115.130:46065)
0924 20:08:01.055351 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:01.055367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:01.055478 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:01.055726 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:01.055728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:01.055734 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:01.055840 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:01.055844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.056789 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.056792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:01.057821 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:01.057832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.058324 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.058327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:01.058410 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:01.058460 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:01.058818 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:01.058824 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:08:01.058879 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 20:08:01.058949 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":146,"threads_started":1}
I20260924 20:08:01.059129 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:01.055080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:46065 (local address 127.0.115.129:37763)
0924 20:08:01.055215 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:01.055219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:01.055445 (+ 226us) server_negotiation.cc:407] Connection header received
0924 20:08:01.055580 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:01.055582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:01.055629 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:01.055699 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:01.056040 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.056675 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.057955 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.058195 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.058287 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:01.058606 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:01.058687 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:01.058750 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:01.058945 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 20:08:01.059005 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:08:01.066121 2363 raft_consensus.cc:1261] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Refusing update from remote peer 4aed5cb0f079488088bdb425f219edd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:01.066684 2643 raft_consensus.cc:1261] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Refusing update from remote peer 4aed5cb0f079488088bdb425f219edd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:01.066790 2719 consensus_queue.cc:1035] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:01.067170 2764 consensus_queue.cc:1035] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:01.085049 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:01.081587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:42355 (local address 127.0.115.129:53111)
0924 20:08:01.081714 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:01.081730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:01.081812 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:01.082306 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:01.082309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:01.082317 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:01.082422 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:01.082428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.083186 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.083190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:01.083782 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:01.083790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.084279 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.084283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:01.084370 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:01.084408 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:01.084753 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:01.084761 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:08:01.084821 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:08:01.084898 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 20:08:01.085070 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:01.081821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:53111 (local address 127.0.115.130:42355)
0924 20:08:01.082080 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:01.082085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:01.082101 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:01.082150 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:01.082153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:01.082202 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:01.082291 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:01.082538 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.083069 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.083904 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:01.084162 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:01.084277 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:01.084537 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:01.084659 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:01.084732 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:01.084844 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 20:08:01.084903 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
W20260924 20:08:01.085783 2739 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 20:08:02.665740 2200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:02.665887 2200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:02.668680 2200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Sending DeleteTablet for 3 replicas of tablet da08319889a44a3e80d4e8f9c5bcdba5
I20260924 20:08:02.669456 2623 tablet_service.cc:1514] Processing DeleteTablet for tablet da08319889a44a3e80d4e8f9c5bcdba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:02 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:33382
I20260924 20:08:02.669446 2482 tablet_service.cc:1514] Processing DeleteTablet for tablet da08319889a44a3e80d4e8f9c5bcdba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:02 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:47522
I20260924 20:08:02.669806 2797 tablet_replica.cc:330] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: stopping tablet replica
I20260924 20:08:02.669795 2343 tablet_service.cc:1514] Processing DeleteTablet for tablet da08319889a44a3e80d4e8f9c5bcdba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:02 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:56472
I20260924 20:08:02.669942 2797 raft_consensus.cc:2229] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:02.670152 2798 tablet_replica.cc:330] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: stopping tablet replica
I20260924 20:08:02.670359 2797 raft_consensus.cc:2258] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:02.670351 2798 raft_consensus.cc:2229] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:02.670500 2798 raft_consensus.cc:2258] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:02.671195 2797 ts_tablet_manager.cc:1905] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:02.673337 2798 ts_tablet_manager.cc:1905] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:02.673561 2797 ts_tablet_manager.cc:1918] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:08:02.673655 2797 log.cc:1196] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/da08319889a44a3e80d4e8f9c5bcdba5
I20260924 20:08:02.673857 2796 tablet_replica.cc:330] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: stopping tablet replica
I20260924 20:08:02.674000 2796 raft_consensus.cc:2229] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:02.674177 2796 raft_consensus.cc:2258] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:02.674624 2797 ts_tablet_manager.cc:1939] T da08319889a44a3e80d4e8f9c5bcdba5 P 4aed5cb0f079488088bdb425f219edd6: Deleting consensus metadata
I20260924 20:08:02.675594 2796 ts_tablet_manager.cc:1905] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:02.676201 2192 catalog_manager.cc:4841] TS 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355): tablet da08319889a44a3e80d4e8f9c5bcdba5 (table test-table [id=6b85541126d44ddaaae472f2f82f5884]) successfully deleted
I20260924 20:08:02.676776 2798 ts_tablet_manager.cc:1918] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:08:02.676855 2798 log.cc:1196] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/da08319889a44a3e80d4e8f9c5bcdba5
I20260924 20:08:02.677140 2798 ts_tablet_manager.cc:1939] T da08319889a44a3e80d4e8f9c5bcdba5 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting consensus metadata
I20260924 20:08:02.677970 2193 catalog_manager.cc:4841] TS fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763): tablet da08319889a44a3e80d4e8f9c5bcdba5 (table test-table [id=6b85541126d44ddaaae472f2f82f5884]) successfully deleted
I20260924 20:08:02.678038 2796 ts_tablet_manager.cc:1918] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:08:02.678092 2796 log.cc:1196] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/da08319889a44a3e80d4e8f9c5bcdba5
I20260924 20:08:02.678309 2796 ts_tablet_manager.cc:1939] T da08319889a44a3e80d4e8f9c5bcdba5 P b8f45831ae88444e959f0b7cb6803850: Deleting consensus metadata
I20260924 20:08:02.679066 2192 catalog_manager.cc:4841] TS b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343): tablet da08319889a44a3e80d4e8f9c5bcdba5 (table test-table [id=6b85541126d44ddaaae472f2f82f5884]) successfully deleted
Sep 24 20:08:02 dist-test-slave-2pm7 krb5kdc[2161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:08:02 dist-test-slave-2pm7 krb5kdc[2161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:02.704352 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:02.691475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.0.115.190:41051)
0924 20:08:02.691779 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:02.691786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:02.691799 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:02.691842 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:02.691844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:02.691884 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:02.692029 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:02.692682 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:02.693219 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:02.694018 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:02.694248 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:02.694326 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:02.701205 (+ 6879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:02.701208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:02.701210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:02.701237 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:02.702618 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:02.703006 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:02.703009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:02.703010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:02.703099 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:02.703393 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:02.703396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:02.703398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:02.703602 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:02.703716 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:02.704102 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 20:08:02.704214 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":147,"threads_started":1}
I20260924 20:08:02.707139 2200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:02.707653 2200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:02.712677 2343 tablet_service.cc:1467] Processing CreateTablet for tablet efc17cd60d61411384e1a7c952d28732 (DEFAULT_TABLE table=test-table [id=514878c8dad1483cbeca60793add6d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:02.712709 2482 tablet_service.cc:1467] Processing CreateTablet for tablet efc17cd60d61411384e1a7c952d28732 (DEFAULT_TABLE table=test-table [id=514878c8dad1483cbeca60793add6d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:02.712852 2623 tablet_service.cc:1467] Processing CreateTablet for tablet efc17cd60d61411384e1a7c952d28732 (DEFAULT_TABLE table=test-table [id=514878c8dad1483cbeca60793add6d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:02.712990 2343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc17cd60d61411384e1a7c952d28732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:02.713045 2482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc17cd60d61411384e1a7c952d28732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:02.713097 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc17cd60d61411384e1a7c952d28732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:02.715768 2807 tablet_bootstrap.cc:492] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: Bootstrap starting.
I20260924 20:08:02.715996 2809 tablet_bootstrap.cc:492] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: Bootstrap starting.
I20260924 20:08:02.716650 2807 tablet_bootstrap.cc:654] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:02.716771 2809 tablet_bootstrap.cc:654] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:02.717376 2808 tablet_bootstrap.cc:492] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: Bootstrap starting.
I20260924 20:08:02.717895 2807 tablet_bootstrap.cc:492] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: No bootstrap required, opened a new log
I20260924 20:08:02.718014 2809 tablet_bootstrap.cc:492] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: No bootstrap required, opened a new log
I20260924 20:08:02.718044 2807 ts_tablet_manager.cc:1397] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:02.718142 2809 ts_tablet_manager.cc:1397] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:02.718308 2808 tablet_bootstrap.cc:654] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:02.718616 2807 raft_consensus.cc:348] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.718761 2807 raft_consensus.cc:374] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:02.718820 2807 raft_consensus.cc:729] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Initialized, Role: FOLLOWER
I20260924 20:08:02.718962 2807 consensus_queue.cc:260] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.719246 2809 raft_consensus.cc:348] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.719332 2807 ts_tablet_manager.cc:1428] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:02.719424 2809 raft_consensus.cc:374] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:02.719460 2808 tablet_bootstrap.cc:492] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: No bootstrap required, opened a new log
I20260924 20:08:02.719515 2809 raft_consensus.cc:729] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f45831ae88444e959f0b7cb6803850, State: Initialized, Role: FOLLOWER
I20260924 20:08:02.719558 2808 ts_tablet_manager.cc:1397] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:02.719660 2809 consensus_queue.cc:260] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.720039 2809 ts_tablet_manager.cc:1428] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:02.720041 2808 raft_consensus.cc:348] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.720163 2808 raft_consensus.cc:374] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:02.720197 2808 raft_consensus.cc:729] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c8eacb0ac4f3689406c71f319f2bb, State: Initialized, Role: FOLLOWER
I20260924 20:08:02.720320 2808 consensus_queue.cc:260] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.720597 2808 ts_tablet_manager.cc:1428] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:02.755406 2787 raft_consensus.cc:482] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:02.755548 2787 raft_consensus.cc:504] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.756151 2787 leader_election.cc:288] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:02.756649 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc17cd60d61411384e1a7c952d28732" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" is_pre_election: true
I20260924 20:08:02.756697 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc17cd60d61411384e1a7c952d28732" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850" is_pre_election: true
I20260924 20:08:02.756830 2643 raft_consensus.cc:2454] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 0.
I20260924 20:08:02.756824 2362 raft_consensus.cc:2454] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 0.
I20260924 20:08:02.757133 2437 leader_election.cc:302] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, fc5c8eacb0ac4f3689406c71f319f2bb; no voters:
I20260924 20:08:02.757339 2787 raft_consensus.cc:2781] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:02.757398 2787 raft_consensus.cc:482] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:02.757463 2787 raft_consensus.cc:3037] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:02.758503 2787 raft_consensus.cc:504] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.758845 2787 leader_election.cc:288] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 election: Requested vote from peers fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:02.759281 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc17cd60d61411384e1a7c952d28732" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb"
I20260924 20:08:02.759296 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc17cd60d61411384e1a7c952d28732" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850"
I20260924 20:08:02.759398 2362 raft_consensus.cc:3037] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:02.759428 2643 raft_consensus.cc:3037] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:02.760386 2643 raft_consensus.cc:2454] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 1.
I20260924 20:08:02.760457 2362 raft_consensus.cc:2454] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 1.
I20260924 20:08:02.760739 2436 leader_election.cc:302] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, b8f45831ae88444e959f0b7cb6803850; no voters:
I20260924 20:08:02.760912 2787 raft_consensus.cc:2781] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:02.761013 2787 raft_consensus.cc:686] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 1 LEADER]: Becoming Leader. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Running, Role: LEADER
I20260924 20:08:02.761165 2787 consensus_queue.cc:237] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.762887 2206 catalog_manager.cc:5495] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130). New cstate: current_term: 1 leader_uuid: "4aed5cb0f079488088bdb425f219edd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:02.788745 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:02.785547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.0.115.130:42355)
0924 20:08:02.785797 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:02.785813 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:02.785831 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:08:02.785914 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:02.785918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:02.785971 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:02.786056 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:02.786285 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:02.786798 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:02.787701 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:02.787925 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:02.788058 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:02.788289 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:02.788358 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:02.788412 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:02.788571 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 20:08:02.788629 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
I20260924 20:08:02.794669 2643 raft_consensus.cc:1261] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Refusing update from remote peer 4aed5cb0f079488088bdb425f219edd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:02.794723 2362 raft_consensus.cc:1261] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Refusing update from remote peer 4aed5cb0f079488088bdb425f219edd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:02.795145 2790 consensus_queue.cc:1035] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:02.795351 2787 consensus_queue.cc:1035] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:02.827711 2206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:02.827904 2206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:02.829991 2206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Sending DeleteTablet for 3 replicas of tablet efc17cd60d61411384e1a7c952d28732
I20260924 20:08:02.830629 2482 tablet_service.cc:1514] Processing DeleteTablet for tablet efc17cd60d61411384e1a7c952d28732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:02 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:47522
I20260924 20:08:02.830827 2797 tablet_replica.cc:330] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: stopping tablet replica
I20260924 20:08:02.830827 2623 tablet_service.cc:1514] Processing DeleteTablet for tablet efc17cd60d61411384e1a7c952d28732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:02 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:33382
I20260924 20:08:02.830955 2797 raft_consensus.cc:2229] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:02.831025 2796 tablet_replica.cc:330] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: stopping tablet replica
I20260924 20:08:02.831070 2343 tablet_service.cc:1514] Processing DeleteTablet for tablet efc17cd60d61411384e1a7c952d28732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:02 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:56472
I20260924 20:08:02.831151 2796 raft_consensus.cc:2229] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:02.831178 2797 raft_consensus.cc:2258] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:02.831259 2798 tablet_replica.cc:330] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: stopping tablet replica
I20260924 20:08:02.831287 2796 raft_consensus.cc:2258] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:02.831370 2798 raft_consensus.cc:2229] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:02.831480 2798 raft_consensus.cc:2258] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:02.831367 2206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:08:02.831918 2797 ts_tablet_manager.cc:1905] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:08:02.831991 2206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:02.832194 2796 ts_tablet_manager.cc:1905] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:02.832422 2798 ts_tablet_manager.cc:1905] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:02.834690 2797 ts_tablet_manager.cc:1918] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:02.834745 2797 log.cc:1196] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/efc17cd60d61411384e1a7c952d28732
I20260924 20:08:02.834828 2796 ts_tablet_manager.cc:1918] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:02.834905 2796 log.cc:1196] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/efc17cd60d61411384e1a7c952d28732
I20260924 20:08:02.834929 2797 ts_tablet_manager.cc:1939] T efc17cd60d61411384e1a7c952d28732 P 4aed5cb0f079488088bdb425f219edd6: Deleting consensus metadata
I20260924 20:08:02.834968 2798 ts_tablet_manager.cc:1918] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:02.835036 2798 log.cc:1196] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/efc17cd60d61411384e1a7c952d28732
I20260924 20:08:02.835175 2796 ts_tablet_manager.cc:1939] T efc17cd60d61411384e1a7c952d28732 P b8f45831ae88444e959f0b7cb6803850: Deleting consensus metadata
I20260924 20:08:02.835335 2798 ts_tablet_manager.cc:1939] T efc17cd60d61411384e1a7c952d28732 P fc5c8eacb0ac4f3689406c71f319f2bb: Deleting consensus metadata
I20260924 20:08:02.835932 2192 catalog_manager.cc:4841] TS b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343): tablet efc17cd60d61411384e1a7c952d28732 (table test-table [id=514878c8dad1483cbeca60793add6d7c]) successfully deleted
I20260924 20:08:02.836012 2193 catalog_manager.cc:4841] TS fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763): tablet efc17cd60d61411384e1a7c952d28732 (table test-table [id=514878c8dad1483cbeca60793add6d7c]) successfully deleted
I20260924 20:08:02.836161 2192 catalog_manager.cc:4841] TS 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355): tablet efc17cd60d61411384e1a7c952d28732 (table test-table [id=514878c8dad1483cbeca60793add6d7c]) successfully deleted
I20260924 20:08:02.839041 2343 tablet_service.cc:1467] Processing CreateTablet for tablet 9fca5dded3ff4204a2b78497de17899a (DEFAULT_TABLE table=test-table [id=6542d01d74e043dfbe496b308012c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:02.839076 2482 tablet_service.cc:1467] Processing CreateTablet for tablet 9fca5dded3ff4204a2b78497de17899a (DEFAULT_TABLE table=test-table [id=6542d01d74e043dfbe496b308012c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:02.839322 2343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fca5dded3ff4204a2b78497de17899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:02.839349 2482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fca5dded3ff4204a2b78497de17899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:02.839426 2623 tablet_service.cc:1467] Processing CreateTablet for tablet 9fca5dded3ff4204a2b78497de17899a (DEFAULT_TABLE table=test-table [id=6542d01d74e043dfbe496b308012c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:02.839653 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fca5dded3ff4204a2b78497de17899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:02.841192 2807 tablet_bootstrap.cc:492] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: Bootstrap starting.
I20260924 20:08:02.841810 2809 tablet_bootstrap.cc:492] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: Bootstrap starting.
I20260924 20:08:02.841820 2808 tablet_bootstrap.cc:492] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb: Bootstrap starting.
I20260924 20:08:02.841980 2807 tablet_bootstrap.cc:654] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:02.842607 2808 tablet_bootstrap.cc:654] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:02.842763 2809 tablet_bootstrap.cc:654] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:02.843163 2807 tablet_bootstrap.cc:492] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: No bootstrap required, opened a new log
I20260924 20:08:02.843269 2807 ts_tablet_manager.cc:1397] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:02.843720 2807 raft_consensus.cc:348] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.843814 2808 tablet_bootstrap.cc:492] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb: No bootstrap required, opened a new log
I20260924 20:08:02.843839 2807 raft_consensus.cc:374] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:02.843873 2807 raft_consensus.cc:729] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Initialized, Role: FOLLOWER
I20260924 20:08:02.843891 2808 ts_tablet_manager.cc:1397] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:02.844034 2807 consensus_queue.cc:260] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.844312 2807 ts_tablet_manager.cc:1428] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:08:02.844324 2808 raft_consensus.cc:348] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.844421 2808 raft_consensus.cc:374] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:02.844455 2808 raft_consensus.cc:729] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c8eacb0ac4f3689406c71f319f2bb, State: Initialized, Role: FOLLOWER
I20260924 20:08:02.844579 2808 consensus_queue.cc:260] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.844851 2808 ts_tablet_manager.cc:1428] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:02.845095 2809 tablet_bootstrap.cc:492] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: No bootstrap required, opened a new log
I20260924 20:08:02.845193 2809 ts_tablet_manager.cc:1397] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:02.845691 2809 raft_consensus.cc:348] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.845805 2809 raft_consensus.cc:374] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:02.845844 2809 raft_consensus.cc:729] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f45831ae88444e959f0b7cb6803850, State: Initialized, Role: FOLLOWER
I20260924 20:08:02.845963 2809 consensus_queue.cc:260] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.846231 2809 ts_tablet_manager.cc:1428] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:02.926752 2772 raft_consensus.cc:482] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:02.926908 2772 raft_consensus.cc:504] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.927302 2772 leader_election.cc:288] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:02.927891 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca5dded3ff4204a2b78497de17899a" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850" is_pre_election: true
I20260924 20:08:02.927870 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca5dded3ff4204a2b78497de17899a" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" is_pre_election: true
I20260924 20:08:02.928066 2362 raft_consensus.cc:2454] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 0.
I20260924 20:08:02.928067 2642 raft_consensus.cc:2454] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 0.
I20260924 20:08:02.928365 2436 leader_election.cc:302] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, b8f45831ae88444e959f0b7cb6803850; no voters:
I20260924 20:08:02.928589 2772 raft_consensus.cc:2781] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:02.928658 2772 raft_consensus.cc:482] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:02.928726 2772 raft_consensus.cc:3037] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:02.929714 2772 raft_consensus.cc:504] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.930094 2772 leader_election.cc:288] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 election: Requested vote from peers fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763), b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343)
I20260924 20:08:02.930461 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca5dded3ff4204a2b78497de17899a" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f45831ae88444e959f0b7cb6803850"
I20260924 20:08:02.930469 2362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca5dded3ff4204a2b78497de17899a" candidate_uuid: "4aed5cb0f079488088bdb425f219edd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb"
I20260924 20:08:02.930569 2642 raft_consensus.cc:3037] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:02.930634 2362 raft_consensus.cc:3037] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:02.931547 2642 raft_consensus.cc:2454] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 1.
I20260924 20:08:02.931592 2362 raft_consensus.cc:2454] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aed5cb0f079488088bdb425f219edd6 in term 1.
I20260924 20:08:02.931919 2436 leader_election.cc:302] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4aed5cb0f079488088bdb425f219edd6, b8f45831ae88444e959f0b7cb6803850, fc5c8eacb0ac4f3689406c71f319f2bb; no voters:
I20260924 20:08:02.932119 2772 raft_consensus.cc:2781] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:02.932260 2772 raft_consensus.cc:686] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 1 LEADER]: Becoming Leader. State: Replica: 4aed5cb0f079488088bdb425f219edd6, State: Running, Role: LEADER
I20260924 20:08:02.932410 2772 consensus_queue.cc:237] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } }
I20260924 20:08:02.934075 2205 catalog_manager.cc:5495] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130). New cstate: current_term: 1 leader_uuid: "4aed5cb0f079488088bdb425f219edd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aed5cb0f079488088bdb425f219edd6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:02.953063 2362 raft_consensus.cc:1261] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Refusing update from remote peer 4aed5cb0f079488088bdb425f219edd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:02.952828 2642 raft_consensus.cc:1261] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Refusing update from remote peer 4aed5cb0f079488088bdb425f219edd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:02.953581 2772 consensus_queue.cc:1035] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5c8eacb0ac4f3689406c71f319f2bb" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:02.953779 2790 consensus_queue.cc:1035] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f45831ae88444e959f0b7cb6803850" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:08:02.967903 2777 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 20:08:04.518570 2205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:04.518731 2205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:04.521307 2205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d01698238d4e7a96d78ceb19f1506a: Sending DeleteTablet for 3 replicas of tablet 9fca5dded3ff4204a2b78497de17899a
I20260924 20:08:04.521934 2343 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fca5dded3ff4204a2b78497de17899a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:04 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:56472
I20260924 20:08:04.521942 2482 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fca5dded3ff4204a2b78497de17899a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:04 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:47522
I20260924 20:08:04.522073 2623 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fca5dded3ff4204a2b78497de17899a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:04 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:33382
I20260924 20:08:04.522344 2848 tablet_replica.cc:330] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: stopping tablet replica
I20260924 20:08:04.522344 2847 tablet_replica.cc:330] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb: stopping tablet replica
I20260924 20:08:04.522480 2848 raft_consensus.cc:2229] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:04.522480 2847 raft_consensus.cc:2229] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:04.522593 2847 raft_consensus.cc:2258] T 9fca5dded3ff4204a2b78497de17899a P fc5c8eacb0ac4f3689406c71f319f2bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:04.522675 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 2278
I20260924 20:08:04.522693 2848 raft_consensus.cc:2258] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:04.523820 2849 tablet_replica.cc:330] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: stopping tablet replica
I20260924 20:08:04.523990 2849 raft_consensus.cc:2229] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:04.524107 2849 raft_consensus.cc:2258] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:04.524287 2848 ts_tablet_manager.cc:1905] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:04.524741 2849 ts_tablet_manager.cc:1905] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:04.527067 2848 ts_tablet_manager.cc:1918] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:08:04.527153 2848 log.cc:1196] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/9fca5dded3ff4204a2b78497de17899a
I20260924 20:08:04.527355 2849 ts_tablet_manager.cc:1918] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:08:04.527416 2848 ts_tablet_manager.cc:1939] T 9fca5dded3ff4204a2b78497de17899a P 4aed5cb0f079488088bdb425f219edd6: Deleting consensus metadata
I20260924 20:08:04.527431 2849 log.cc:1196] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/9fca5dded3ff4204a2b78497de17899a
I20260924 20:08:04.527693 2849 ts_tablet_manager.cc:1939] T 9fca5dded3ff4204a2b78497de17899a P b8f45831ae88444e959f0b7cb6803850: Deleting consensus metadata
I20260924 20:08:04.528527 2192 catalog_manager.cc:4841] TS b8f45831ae88444e959f0b7cb6803850 (127.0.115.131:42343): tablet 9fca5dded3ff4204a2b78497de17899a (table test-table [id=6542d01d74e043dfbe496b308012c255]) successfully deleted
I20260924 20:08:04.528848 2192 catalog_manager.cc:4841] TS 4aed5cb0f079488088bdb425f219edd6 (127.0.115.130:42355): tablet 9fca5dded3ff4204a2b78497de17899a (table test-table [id=6542d01d74e043dfbe496b308012c255]) successfully deleted
W20260924 20:08:04.531616 2193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:37763 (error 108)
I20260924 20:08:04.531848 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 2418
I20260924 20:08:04.533538 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:04.532996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:37763 (local address 127.0.0.1:60290)
0924 20:08:04.533292 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:04.533384 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:37763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
W20260924 20:08:04.533823 2193 catalog_manager.cc:4584] TS fc5c8eacb0ac4f3689406c71f319f2bb (127.0.115.129:37763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fca5dded3ff4204a2b78497de17899a: Network error: Client connection negotiation failed: client connection to 127.0.115.129:37763: connect: Connection refused (error 111)
I20260924 20:08:04.539407 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 2558
I20260924 20:08:04.545222 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 2174
2026-09-24T20:08:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6547 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:04 dist-test-slave-2pm7 krb5kdc[2854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:04 dist-test-slave-2pm7 krb5kdc[2854](info): set up 2 sockets
Sep 24 20:08:04 dist-test-slave-2pm7 krb5kdc[2854](info): commencing operation
krb5kdc: starting...
W20260924 20:08:05.584213 462 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:05 dist-test-slave-2pm7 krb5kdc[2854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:05.711119 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:40241
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:05.823295 2867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:05.823566 2867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:05.823607 2867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:05.827632 2867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:05.827694 2867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:05.827713 2867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:05.827729 2867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:05.827744 2867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:05.833173 2867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:40241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:05.834251 2867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:05.835223 2867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:05.844225 2875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:05.844280 2867 server_base.cc:1034] running on GCE node
W20260924 20:08:05.844208 2872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:05.844236 2873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:05.845002 2867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:05.846120 2867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:05.847316 2867 hybrid_clock.cc:648] HybridClock initialized: now 1790280485847296 us; error 31 us; skew 500 ppm
Sep 24 20:08:05 dist-test-slave-2pm7 krb5kdc[2854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:05.850301 2867 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:05.851296 2867 webserver.cc:460] Webserver started at http://127.0.115.190:44249/ using document root <none> and password file <none>
I20260924 20:08:05.851933 2867 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:05.852092 2867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:05.852342 2867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:05.854593 2867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "58cc54cd81254818bf516459f616c2a4"
format_stamp: "Formatted at 2026-09-24 20:08:05 on dist-test-slave-2pm7"
server_key: "5227159f536dbc84d28f638dfd7da52a"
server_key_iv: "b1ecfb5e71425107ade5c9da72c32d58"
server_key_version: "encryptionkey@0"
I20260924 20:08:05.855232 2867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "58cc54cd81254818bf516459f616c2a4"
format_stamp: "Formatted at 2026-09-24 20:08:05 on dist-test-slave-2pm7"
server_key: "5227159f536dbc84d28f638dfd7da52a"
server_key_iv: "b1ecfb5e71425107ade5c9da72c32d58"
server_key_version: "encryptionkey@0"
I20260924 20:08:05.859326 2867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:08:05.862030 2882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:05.863329 2867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:05.863493 2867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "58cc54cd81254818bf516459f616c2a4"
format_stamp: "Formatted at 2026-09-24 20:08:05 on dist-test-slave-2pm7"
server_key: "5227159f536dbc84d28f638dfd7da52a"
server_key_iv: "b1ecfb5e71425107ade5c9da72c32d58"
server_key_version: "encryptionkey@0"
I20260924 20:08:05.863639 2867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:05.874861 2867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:05.878480 2867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:05.878701 2867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:05.886567 2867 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:40241
I20260924 20:08:05.886631 2934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:40241 every 8 connection(s)
I20260924 20:08:05.887698 2867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:05.890568 2935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:05.896507 2935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4: Bootstrap starting.
I20260924 20:08:05.897804 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 2867
I20260924 20:08:05.897939 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:05.898197 462 external_mini_cluster.cc:1428] Setting key 780d3fb5794796aef8a549a7d7578f00
I20260924 20:08:05.899291 2935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:05.900184 2935 log.cc:824] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:05.902480 2935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4: No bootstrap required, opened a new log
I20260924 20:08:05.907016 2935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc54cd81254818bf516459f616c2a4" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40241 } }
I20260924 20:08:05.907318 2935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:05.907401 2935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58cc54cd81254818bf516459f616c2a4, State: Initialized, Role: FOLLOWER
Sep 24 20:08:05 dist-test-slave-2pm7 krb5kdc[2854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:05.907936 2935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc54cd81254818bf516459f616c2a4" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40241 } }
I20260924 20:08:05.908149 2935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:05.908227 2935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:05.908332 2935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:05.909268 2935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc54cd81254818bf516459f616c2a4" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40241 } }
I20260924 20:08:05.909632 2935 leader_election.cc:302] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58cc54cd81254818bf516459f616c2a4; no voters:
I20260924 20:08:05.909942 2935 leader_election.cc:288] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:05.910065 2940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:05.910280 2940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 58cc54cd81254818bf516459f616c2a4, State: Running, Role: LEADER
I20260924 20:08:05.910663 2940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc54cd81254818bf516459f616c2a4" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40241 } }
I20260924 20:08:05.911367 2935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:05.912487 2941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58cc54cd81254818bf516459f616c2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc54cd81254818bf516459f616c2a4" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40241 } } }
I20260924 20:08:05.912626 2941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:05.912940 2942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58cc54cd81254818bf516459f616c2a4. Latest consensus state: current_term: 1 leader_uuid: "58cc54cd81254818bf516459f616c2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc54cd81254818bf516459f616c2a4" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40241 } } }
I20260924 20:08:05.913038 2942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:05.914069 2949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:05.914754 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:05.899345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.0.115.190:40241)
0924 20:08:05.899824 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:05.899842 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:05.899879 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:08:05.900642 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:05.900660 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:05.901004 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:05.901294 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:05.902223 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:05.903269 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:05.904519 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:05.905027 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:05.905170 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:05.908473 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:05.908479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:05.908496 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:05.908532 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:05.910646 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:05.911183 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:05.911189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:05.911195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:05.911339 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:05.911649 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:05.911652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:05.911654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:05.912260 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:05.912468 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:05.913425 (+ 957us) server_negotiation.cc:299] Negotiation successful
0924 20:08:05.913786 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260924 20:08:05.917452 2949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:05.923623 2949 catalog_manager.cc:1347] Generated new cluster ID: 059f755d3bfe4dd5accfd2453a7ea0bf
I20260924 20:08:05.923707 2949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:05.945819 2949 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:05.946839 2949 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:05.955302 2949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58cc54cd81254818bf516459f616c2a4: Generated new TSK 0
I20260924 20:08:05.956102 2949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:08:05.984812 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40241
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:06.098781 2961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:06.099107 2961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:06.099188 2961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:06.103343 2961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:06.103440 2961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:06.103536 2961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:06.108963 2961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:06.110060 2961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:06.111495 2961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:06.121578 2966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:06.121562 2969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:06.121683 2967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:06.121795 2961 server_base.cc:1034] running on GCE node
I20260924 20:08:06.122284 2961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:06.122951 2961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:06.124111 2961 hybrid_clock.cc:648] HybridClock initialized: now 1790280486124091 us; error 30 us; skew 500 ppm
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[2854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:06.126989 2961 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:08:06.127980 2961 webserver.cc:460] Webserver started at http://127.0.115.129:44769/ using document root <none> and password file <none>
I20260924 20:08:06.128561 2961 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:06.128628 2961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:06.128865 2961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:06.131141 2961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "9213c31ad2124f4f925b7591c31c42ba"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "02e9bdc1bdafdc322936e5d4ef8bd632"
server_key_iv: "fbd682446252b43bb6d84fbc8b27c0e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.131697 2961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "9213c31ad2124f4f925b7591c31c42ba"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "02e9bdc1bdafdc322936e5d4ef8bd632"
server_key_iv: "fbd682446252b43bb6d84fbc8b27c0e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.135437 2961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:06.137965 2976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.139180 2961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:06.139322 2961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "9213c31ad2124f4f925b7591c31c42ba"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "02e9bdc1bdafdc322936e5d4ef8bd632"
server_key_iv: "fbd682446252b43bb6d84fbc8b27c0e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.139433 2961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:06.149638 2961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:06.152510 2961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:06.152717 2961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:06.153326 2961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:06.154155 2961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:06.154227 2961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.154295 2961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:06.154345 2961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.163417 2961 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:40027
I20260924 20:08:06.163439 3089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:40027 every 8 connection(s)
I20260924 20:08:06.164361 2961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:06.171690 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 2961
I20260924 20:08:06.171804 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:06.172111 462 external_mini_cluster.cc:1428] Setting key 28c397eb9785f618031ccffec5a1fc18
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[2854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:06.178998 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:06.166198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:39107 (local address 127.0.115.190:40241)
0924 20:08:06.166335 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:06.166339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:06.167202 (+ 863us) server_negotiation.cc:407] Connection header received
0924 20:08:06.168830 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:06.168833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:06.168891 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:06.168969 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:06.170104 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.170643 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.172002 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.172309 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.172405 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:06.175065 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:06.175072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:06.175077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:06.175112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:06.176649 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:06.177293 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:06.177295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:06.177297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:06.177369 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:06.177796 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:06.177798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:06.177799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:06.177980 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:06.178056 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:06.178759 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 20:08:06.178873 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:08:06.180367 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:06.166589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:40241 (local address 127.0.115.129:39107)
0924 20:08:06.167049 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:06.167090 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:06.168615 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:06.169126 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:06.169132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:06.169471 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:06.169968 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:06.169981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.170758 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.170761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:06.171821 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:06.171832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.172476 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.172481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:06.172659 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:06.173282 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:06.173299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:06.174874 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:06.176777 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:06.176787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:06.176805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:06.177197 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:06.177463 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:06.177470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:06.177475 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:06.177683 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:06.178100 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:06.178110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:06.178469 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 20:08:06.178720 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 20:08:06.179140 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260924 20:08:06.181740 3090 heartbeater.cc:347] Connected to a master server at 127.0.115.190:40241
I20260924 20:08:06.181991 3090 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:08:06.182520 3090 heartbeater.cc:511] Master 127.0.115.190:40241 requested a full tablet report, sending...
I20260924 20:08:06.184151 2899 ts_manager.cc:194] Registered new tserver with Master: 9213c31ad2124f4f925b7591c31c42ba (127.0.115.129:40027)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:08:06.185576 2899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:39107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:08:06.201025 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40241
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:06.315109 3095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:06.315404 3095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:06.315475 3095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:06.319514 3095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:06.319607 3095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:06.319697 3095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:06.324890 3095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:06.326026 3095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:06.327057 3095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:06.336683 3100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:06.336704 3103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:06.336683 3101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:06.337109 3095 server_base.cc:1034] running on GCE node
I20260924 20:08:06.337633 3095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:06.338550 3095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:06.339763 3095 hybrid_clock.cc:648] HybridClock initialized: now 1790280486339726 us; error 51 us; skew 500 ppm
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[2854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:06.343729 3095 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:08:06.344972 3095 webserver.cc:460] Webserver started at http://127.0.115.130:42083/ using document root <none> and password file <none>
I20260924 20:08:06.345666 3095 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:06.345732 3095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:06.345906 3095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:06.348102 3095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "5f76af1bb53f486a988e58a99c53d2c8"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "9dd3d35fa24b62e1cc93857b9342b882"
server_key_iv: "2395ed29631534d51dc9aec7fb6d8f71"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.348644 3095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "5f76af1bb53f486a988e58a99c53d2c8"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "9dd3d35fa24b62e1cc93857b9342b882"
server_key_iv: "2395ed29631534d51dc9aec7fb6d8f71"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.352355 3095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:08:06.354864 3110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.356323 3095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:06.356453 3095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "5f76af1bb53f486a988e58a99c53d2c8"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "9dd3d35fa24b62e1cc93857b9342b882"
server_key_iv: "2395ed29631534d51dc9aec7fb6d8f71"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.356563 3095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:06.383886 3095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:06.387513 3095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:06.387739 3095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:06.388441 3095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:06.389351 3095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:06.389398 3095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.389469 3095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:06.389509 3095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.399060 3095 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:45085
I20260924 20:08:06.399085 3223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:45085 every 8 connection(s)
I20260924 20:08:06.400060 3095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:06.407536 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 3095
I20260924 20:08:06.407684 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:06.408028 462 external_mini_cluster.cc:1428] Setting key b7f9f975886148cbe6b9af51b96892a8
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[2854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:06.414665 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:06.401975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:37161 (local address 127.0.115.190:40241)
0924 20:08:06.402116 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:06.402120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:06.403010 (+ 890us) server_negotiation.cc:407] Connection header received
0924 20:08:06.404446 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:06.404449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:06.404498 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:06.404564 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:06.405695 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.406234 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.407321 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.407535 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.407712 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:06.410397 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:06.410403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:06.410410 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:06.410442 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:06.411905 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:06.412725 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:06.412729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:06.412731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:06.412820 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:06.413304 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:06.413309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:06.413312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:06.413597 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:06.413660 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:06.414328 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 20:08:06.414472 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:08:06.415545 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:06.402396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:40241 (local address 127.0.115.130:37161)
0924 20:08:06.402856 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:06.402899 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:06.404230 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:06.404749 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:06.404756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:06.405097 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:06.405565 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:06.405578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.406338 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.406340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:06.407218 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:06.407227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.407757 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.407762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:06.407990 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:06.408702 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:06.408718 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:06.410193 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:06.412108 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:06.412116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:06.412139 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:06.412516 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:06.412953 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:06.412956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:06.412958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:06.413168 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:06.413730 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:06.413735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:06.414037 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 20:08:06.414251 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 20:08:06.414568 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260924 20:08:06.416790 3224 heartbeater.cc:347] Connected to a master server at 127.0.115.190:40241
I20260924 20:08:06.417024 3224 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:06.417520 3224 heartbeater.cc:511] Master 127.0.115.190:40241 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:08:06.418610 2899 ts_manager.cc:194] Registered new tserver with Master: 5f76af1bb53f486a988e58a99c53d2c8 (127.0.115.130:45085)
I20260924 20:08:06.419248 2899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:37161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:08:06.435402 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40241
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:06.550813 3229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:06.551097 3229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:06.551141 3229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:06.555337 3229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:06.555404 3229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:06.555487 3229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:06.560983 3229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:06.562033 3229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:06.563066 3229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:06.572817 3237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:06.572801 3235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:06.572821 3234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:06.573032 3229 server_base.cc:1034] running on GCE node
I20260924 20:08:06.573592 3229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:06.574255 3229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:06.575448 3229 hybrid_clock.cc:648] HybridClock initialized: now 1790280486575406 us; error 57 us; skew 500 ppm
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[2854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:06.578464 3229 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:06.579417 3229 webserver.cc:460] Webserver started at http://127.0.115.131:42559/ using document root <none> and password file <none>
I20260924 20:08:06.580070 3229 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:06.580163 3229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:06.580399 3229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:06.582439 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "d101c935d33c49758c7700ed1cbf6941"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "e79d9c71ee2b227b0dc5ad89b7f87090"
server_key_iv: "63a60c67f011c8d14338cb7cc5abe11c"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.583009 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "d101c935d33c49758c7700ed1cbf6941"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "e79d9c71ee2b227b0dc5ad89b7f87090"
server_key_iv: "63a60c67f011c8d14338cb7cc5abe11c"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.586982 3229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:08:06.589465 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.590649 3229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:06.590799 3229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "d101c935d33c49758c7700ed1cbf6941"
format_stamp: "Formatted at 2026-09-24 20:08:06 on dist-test-slave-2pm7"
server_key: "e79d9c71ee2b227b0dc5ad89b7f87090"
server_key_iv: "63a60c67f011c8d14338cb7cc5abe11c"
server_key_version: "encryptionkey@0"
I20260924 20:08:06.590909 3229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:06.626767 3229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:06.630119 3229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:06.630327 3229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:06.631093 3229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:06.632232 3229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:06.632314 3229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.632385 3229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:06.632416 3229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:06.642055 3229 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44285
I20260924 20:08:06.642086 3357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44285 every 8 connection(s)
I20260924 20:08:06.643007 3229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:06.652419 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 3229
I20260924 20:08:06.652518 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:06.652799 462 external_mini_cluster.cc:1428] Setting key cdb7b65bc401085127ef87a39dd25aba
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[2854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:06.657490 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:06.645119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:32957 (local address 127.0.115.190:40241)
0924 20:08:06.645268 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:06.645271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:06.646170 (+ 899us) server_negotiation.cc:407] Connection header received
0924 20:08:06.647609 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:06.647613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:06.647664 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:06.647744 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:06.649104 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.649704 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.650811 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.651025 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.651105 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:06.653908 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:06.653912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:06.653914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:06.653941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:06.655352 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:06.655894 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:06.655898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:06.655899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:06.656035 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:06.656383 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:06.656387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:06.656388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:06.656581 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:06.656711 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:06.657210 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 20:08:06.657325 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:08:06.658284 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:06.645502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:40241 (local address 127.0.115.131:32957)
0924 20:08:06.646004 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:06.646049 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:06.647367 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:06.647909 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:06.647916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:06.648371 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:06.648930 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:06.648944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.649821 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.649824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:06.650706 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:06.650715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:06.651153 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:06.651156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:06.651304 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:06.651986 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:06.652011 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:06.653755 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:06.655488 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:06.655494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:06.655509 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:06.655765 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:06.656143 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:06.656146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:06.656148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:06.656285 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:06.656719 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:06.656728 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:06.656972 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 20:08:06.657155 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 20:08:06.657471 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
I20260924 20:08:06.659422 3358 heartbeater.cc:347] Connected to a master server at 127.0.115.190:40241
I20260924 20:08:06.659713 3358 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:06.660247 3358 heartbeater.cc:511] Master 127.0.115.190:40241 requested a full tablet report, sending...
I20260924 20:08:06.661309 2899 ts_manager.cc:194] Registered new tserver with Master: d101c935d33c49758c7700ed1cbf6941 (127.0.115.131:44285)
I20260924 20:08:06.661837 2899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:32957
I20260924 20:08:06.666760 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:08:06.679842 2938 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:06.678297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.0.115.190:40241)
0924 20:08:06.678490 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:06.678496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:06.678654 (+ 158us) server_negotiation.cc:407] Connection header received
0924 20:08:06.678823 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:06.678826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:06.678882 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:06.678971 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:06.679723 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42012: BlockingRecv error: recv got EOF from 127.0.0.1:42012 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 20:08:06.681370 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 2961
I20260924 20:08:06.686988 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 3095
I20260924 20:08:06.692746 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 3229
I20260924 20:08:06.698207 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 2867
2026-09-24T20:08:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2152 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[3370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[3370](info): set up 2 sockets
Sep 24 20:08:06 dist-test-slave-2pm7 krb5kdc[3370](info): commencing operation
krb5kdc: starting...
W20260924 20:08:07.752632 462 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:07 dist-test-slave-2pm7 krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:07.878859 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44045
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:07.993757 3383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:07.994051 3383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:07.994141 3383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:07.998584 3383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:07.998658 3383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:07.998678 3383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:07.998695 3383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:07.998713 3383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:08.004295 3383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:08.005432 3383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:08.006529 3383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:08.015352 3391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:08.015504 3383 server_base.cc:1034] running on GCE node
W20260924 20:08:08.015347 3388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:08.015687 3389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:08.016316 3383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:08.017375 3383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:08.018596 3383 hybrid_clock.cc:648] HybridClock initialized: now 1790280488018562 us; error 41 us; skew 500 ppm
Sep 24 20:08:08 dist-test-slave-2pm7 krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:08.028213 3383 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:08.029265 3383 webserver.cc:460] Webserver started at http://127.0.115.190:40903/ using document root <none> and password file <none>
I20260924 20:08:08.029862 3383 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:08.029955 3383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:08.030205 3383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:08.032387 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "c793609233a54818876c21982a62b96c"
format_stamp: "Formatted at 2026-09-24 20:08:08 on dist-test-slave-2pm7"
server_key: "ea34f4dd225b360868f3c682a3207aec"
server_key_iv: "fd6218d33cfe181b48cc47f83d3951bb"
server_key_version: "encryptionkey@0"
I20260924 20:08:08.032966 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "c793609233a54818876c21982a62b96c"
format_stamp: "Formatted at 2026-09-24 20:08:08 on dist-test-slave-2pm7"
server_key: "ea34f4dd225b360868f3c682a3207aec"
server_key_iv: "fd6218d33cfe181b48cc47f83d3951bb"
server_key_version: "encryptionkey@0"
I20260924 20:08:08.036808 3383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:08:08.039408 3398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:08.040810 3383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:08:08.040962 3383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "c793609233a54818876c21982a62b96c"
format_stamp: "Formatted at 2026-09-24 20:08:08 on dist-test-slave-2pm7"
server_key: "ea34f4dd225b360868f3c682a3207aec"
server_key_iv: "fd6218d33cfe181b48cc47f83d3951bb"
server_key_version: "encryptionkey@0"
I20260924 20:08:08.041095 3383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:08.069273 3383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:08.073134 3383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:08.073348 3383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:08.080693 3383 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44045
I20260924 20:08:08.081010 3450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44045 every 8 connection(s)
I20260924 20:08:08.081696 3383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:08.084476 3451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:08.085925 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 3383
I20260924 20:08:08.086043 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:08.086344 462 external_mini_cluster.cc:1428] Setting key c01edef708711c2242d9eca8890a50c6
I20260924 20:08:08.090991 3451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c: Bootstrap starting.
I20260924 20:08:08.093641 3451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:08.094420 3451 log.cc:824] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:08:08 dist-test-slave-2pm7 krb5kdc[3370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:08.096585 3451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c: No bootstrap required, opened a new log
I20260924 20:08:08.100133 3451 raft_consensus.cc:348] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793609233a54818876c21982a62b96c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44045 } }
I20260924 20:08:08.100395 3451 raft_consensus.cc:374] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:08.100490 3451 raft_consensus.cc:729] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c793609233a54818876c21982a62b96c, State: Initialized, Role: FOLLOWER
I20260924 20:08:08.100867 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:08.087644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.0.115.190:44045)
0924 20:08:08.088069 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:08.088091 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:08.088155 (+ 64us) server_negotiation.cc:407] Connection header received
0924 20:08:08.088779 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:08.088795 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:08.089131 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:08.089401 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:08.090336 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:08.091397 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:08.092585 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:08.093062 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:08.093192 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:08.095827 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:08.095832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:08.095847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:08.095874 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:08.097792 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:08.098270 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:08.098278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:08.098289 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:08.098397 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:08.098683 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:08.098686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:08.098688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:08.099082 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:08.099309 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:08.099556 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 20:08:08.099887 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":126,"threads_started":1}
I20260924 20:08:08.100904 3451 consensus_queue.cc:260] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793609233a54818876c21982a62b96c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44045 } }
I20260924 20:08:08.101104 3451 raft_consensus.cc:388] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:08.101186 3451 raft_consensus.cc:482] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:08.101290 3451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:08.102172 3451 raft_consensus.cc:504] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793609233a54818876c21982a62b96c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44045 } }
I20260924 20:08:08.102505 3451 leader_election.cc:302] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c793609233a54818876c21982a62b96c; no voters:
I20260924 20:08:08.102823 3451 leader_election.cc:288] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:08.102888 3456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:08.103112 3456 raft_consensus.cc:686] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [term 1 LEADER]: Becoming Leader. State: Replica: c793609233a54818876c21982a62b96c, State: Running, Role: LEADER
I20260924 20:08:08.103446 3456 consensus_queue.cc:237] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793609233a54818876c21982a62b96c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44045 } }
I20260924 20:08:08.103674 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:08:08.104171 3451 sys_catalog.cc:564] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:08.104677 3456 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c793609233a54818876c21982a62b96c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793609233a54818876c21982a62b96c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44045 } } }
I20260924 20:08:08.104873 3456 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:08.105080 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c793609233a54818876c21982a62b96c. Latest consensus state: current_term: 1 leader_uuid: "c793609233a54818876c21982a62b96c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c793609233a54818876c21982a62b96c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44045 } } }
I20260924 20:08:08.105211 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:08.105559 3463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:08.108845 3463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:08.114548 3463 catalog_manager.cc:1347] Generated new cluster ID: f319bfb1e6f44ac2b63eaac1da5e4368
I20260924 20:08:08.114620 3463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:08.126303 3463 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:08.127486 3463 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:08.127641 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:08.122161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.0.115.190:44045)
0924 20:08:08.122279 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:08.122284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:08.122300 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:08.122362 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:08.122365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:08.122416 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:08.122508 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:08.123075 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:08.123911 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:08.125012 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:08.125266 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:08.125381 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:08.126690 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:08.126694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:08.126697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:08.127139 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:08.127169 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:08.127186 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:08.127286 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:08.127369 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 20:08:08.127432 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:08:08.137454 3463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c793609233a54818876c21982a62b96c: Generated new TSK 0
I20260924 20:08:08.138341 3463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:08:08.145249 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 3383
2026-09-24T20:08:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1444 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:08 dist-test-slave-2pm7 krb5kdc[3484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:08 dist-test-slave-2pm7 krb5kdc[3484](info): set up 2 sockets
Sep 24 20:08:08 dist-test-slave-2pm7 krb5kdc[3484](info): commencing operation
krb5kdc: starting...
W20260924 20:08:09.188153 462 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:09 dist-test-slave-2pm7 krb5kdc[3484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:09.313221 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44363
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:09.427172 3497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:09.427461 3497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:09.427503 3497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:09.431743 3497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:09.431813 3497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:09.431833 3497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:09.431851 3497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:09.431866 3497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:09.437587 3497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:09.438781 3497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:09.439882 3497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:09.448675 3503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:09.448693 3502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:09.448836 3497 server_base.cc:1034] running on GCE node
W20260924 20:08:09.448688 3505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:09.449540 3497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:09.450712 3497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:09.451891 3497 hybrid_clock.cc:648] HybridClock initialized: now 1790280489451864 us; error 50 us; skew 500 ppm
Sep 24 20:08:09 dist-test-slave-2pm7 krb5kdc[3484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:09.454855 3497 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:09.455772 3497 webserver.cc:460] Webserver started at http://127.0.115.190:43223/ using document root <none> and password file <none>
I20260924 20:08:09.456413 3497 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:09.456506 3497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:09.456763 3497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:09.458994 3497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "038c7cb90d18402ea7fefae73c8a4de7"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "bc50f4977b4c02d9196ea3d0a60aff48"
server_key_iv: "d890facf7e443d87e8823c372933c4e3"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.459607 3497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "038c7cb90d18402ea7fefae73c8a4de7"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "bc50f4977b4c02d9196ea3d0a60aff48"
server_key_iv: "d890facf7e443d87e8823c372933c4e3"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.463443 3497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:08:09.466248 3512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:09.467542 3497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:09.467711 3497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "038c7cb90d18402ea7fefae73c8a4de7"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "bc50f4977b4c02d9196ea3d0a60aff48"
server_key_iv: "d890facf7e443d87e8823c372933c4e3"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.467840 3497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:09.483736 3497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:09.488359 3497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:09.488593 3497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:09.496174 3497 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44363
I20260924 20:08:09.496486 3564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44363 every 8 connection(s)
I20260924 20:08:09.497215 3497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:09.499989 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 3497
I20260924 20:08:09.500106 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:09.500432 462 external_mini_cluster.cc:1428] Setting key 967adebd516628f3334489fa8c20d562
I20260924 20:08:09.500473 3565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:09.507798 3565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7: Bootstrap starting.
Sep 24 20:08:09 dist-test-slave-2pm7 krb5kdc[3484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:09.510728 3565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:09.511492 3565 log.cc:824] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:09.513648 3565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7: No bootstrap required, opened a new log
I20260924 20:08:09.515828 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:09.501899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.0.115.190:44363)
0924 20:08:09.502313 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:09.502340 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:09.502377 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:08:09.503268 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:09.503288 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:09.503669 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:09.504030 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:09.504960 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:09.505962 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:09.507149 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:09.507635 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:09.507765 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:09.510597 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:09.510603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:09.510617 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:09.510649 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:09.512549 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:09.513015 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:09.513021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:09.513026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:09.513142 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:09.513520 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:09.513523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:09.513537 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:09.514183 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:09.514406 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:09.514759 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 20:08:09.515091 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":140,"threads_started":1}
I20260924 20:08:09.517617 3565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038c7cb90d18402ea7fefae73c8a4de7" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44363 } }
I20260924 20:08:09.517863 3565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:09.517956 3565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c7cb90d18402ea7fefae73c8a4de7, State: Initialized, Role: FOLLOWER
I20260924 20:08:09.518405 3565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038c7cb90d18402ea7fefae73c8a4de7" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44363 } }
I20260924 20:08:09.518561 3565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:09.518649 3565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:09.518776 3565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:09.519762 3565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038c7cb90d18402ea7fefae73c8a4de7" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44363 } }
I20260924 20:08:09.520162 3565 leader_election.cc:302] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 038c7cb90d18402ea7fefae73c8a4de7; no voters:
I20260924 20:08:09.520509 3565 leader_election.cc:288] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:09.520639 3570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:09.520897 3570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [term 1 LEADER]: Becoming Leader. State: Replica: 038c7cb90d18402ea7fefae73c8a4de7, State: Running, Role: LEADER
I20260924 20:08:09.521261 3570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038c7cb90d18402ea7fefae73c8a4de7" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44363 } }
I20260924 20:08:09.521901 3565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:09.523186 3572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038c7cb90d18402ea7fefae73c8a4de7. Latest consensus state: current_term: 1 leader_uuid: "038c7cb90d18402ea7fefae73c8a4de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038c7cb90d18402ea7fefae73c8a4de7" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44363 } } }
I20260924 20:08:09.523190 3571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "038c7cb90d18402ea7fefae73c8a4de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038c7cb90d18402ea7fefae73c8a4de7" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44363 } } }
I20260924 20:08:09.523387 3571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:09.523689 3572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:09.523793 3579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:09.526825 3579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:09.533361 3579 catalog_manager.cc:1347] Generated new cluster ID: fd261385bd9943989143e68f64c65040
I20260924 20:08:09.533444 3579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:09.539314 3579 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:09.540347 3579 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:09.548977 3579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 038c7cb90d18402ea7fefae73c8a4de7: Generated new TSK 0
I20260924 20:08:09.549682 3579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:08:09.586813 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44363
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:09.700875 3591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:09.701180 3591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:09.701259 3591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:09.705368 3591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:09.705435 3591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:09.705557 3591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:09.711594 3591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:09.713248 3591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:09.714375 3591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:09.724678 3597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:09.724632 3599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:09.724634 3596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:09.724778 3591 server_base.cc:1034] running on GCE node
I20260924 20:08:09.725461 3591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:09.726186 3591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:09.727360 3591 hybrid_clock.cc:648] HybridClock initialized: now 1790280489727341 us; error 29 us; skew 500 ppm
Sep 24 20:08:09 dist-test-slave-2pm7 krb5kdc[3484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:09.730312 3591 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:08:09.731258 3591 webserver.cc:460] Webserver started at http://127.0.115.129:33291/ using document root <none> and password file <none>
I20260924 20:08:09.731810 3591 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:09.731871 3591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:09.732112 3591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:09.734221 3591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "ebf927749b5e4716a262f7a92990332f"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "86243a8bf980e7180de35ad33dbfe94a"
server_key_iv: "19c1c54b157b9488fde531d2fb99a8b8"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.734735 3591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "ebf927749b5e4716a262f7a92990332f"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "86243a8bf980e7180de35ad33dbfe94a"
server_key_iv: "19c1c54b157b9488fde531d2fb99a8b8"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.738381 3591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 20:08:09.741034 3606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:09.742314 3591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:09.742455 3591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "ebf927749b5e4716a262f7a92990332f"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "86243a8bf980e7180de35ad33dbfe94a"
server_key_iv: "19c1c54b157b9488fde531d2fb99a8b8"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.742564 3591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:09.760244 3591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:09.763827 3591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:09.764071 3591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:09.764727 3591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:09.765635 3591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:09.765682 3591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:09.765751 3591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:09.765782 3591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:09.776100 3591 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:32987
I20260924 20:08:09.776147 3719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:32987 every 8 connection(s)
I20260924 20:08:09.777053 3591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:09.783790 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 3591
I20260924 20:08:09.783893 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:09.784235 462 external_mini_cluster.cc:1428] Setting key ac0e10a1d3aacd3227c970f91795c360
Sep 24 20:08:09 dist-test-slave-2pm7 krb5kdc[3484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:09.792023 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:09.778974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:50197 (local address 127.0.115.190:44363)
0924 20:08:09.779147 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:09.779151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:09.780085 (+ 934us) server_negotiation.cc:407] Connection header received
0924 20:08:09.781548 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:09.781552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:09.781612 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:09.781725 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:09.782929 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:09.783476 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:09.784829 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:09.785068 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:09.785208 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:09.788128 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:09.788132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:09.788134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:09.788160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:09.789856 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:09.790416 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:09.790418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:09.790420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:09.790497 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:09.790865 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:09.790869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:09.790871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:09.791057 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:09.791164 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:09.791691 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 20:08:09.791817 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":77}
I20260924 20:08:09.792867 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:09.779417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44363 (local address 127.0.115.129:50197)
0924 20:08:09.779887 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:09.779932 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:09.781323 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:09.781905 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:09.781912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:09.782284 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:09.782771 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:09.782784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:09.783607 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:09.783612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:09.784697 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:09.784708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:09.785235 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:09.785241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:09.785429 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:09.785998 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:09.786019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:09.787908 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:09.789988 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:09.789993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:09.790007 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:09.790289 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:09.790597 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:09.790600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:09.790602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:09.790758 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:09.791166 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:09.791175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:09.791437 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 20:08:09.791629 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 20:08:09.792001 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:08:09.794268 3720 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44363
I20260924 20:08:09.794555 3720 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:09.795058 3720 heartbeater.cc:511] Master 127.0.115.190:44363 requested a full tablet report, sending...
I20260924 20:08:09.796861 3529 ts_manager.cc:194] Registered new tserver with Master: ebf927749b5e4716a262f7a92990332f (127.0.115.129:32987)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:08:09.798179 3529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:50197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:08:09.814952 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44363
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:09.928282 3725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:09.928529 3725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:09.928573 3725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:09.932695 3725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:09.932746 3725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:09.932803 3725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:09.937887 3725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:09.938836 3725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:09.939775 3725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:09.949791 3725 server_base.cc:1034] running on GCE node
W20260924 20:08:09.949806 3731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:09.949816 3733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:09.949965 3730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:09.950481 3725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:09.951148 3725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:09.952332 3725 hybrid_clock.cc:648] HybridClock initialized: now 1790280489952312 us; error 30 us; skew 500 ppm
Sep 24 20:08:09 dist-test-slave-2pm7 krb5kdc[3484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:09.955214 3725 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:08:09.956149 3725 webserver.cc:460] Webserver started at http://127.0.115.130:45599/ using document root <none> and password file <none>
I20260924 20:08:09.956714 3725 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:09.956775 3725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:09.956946 3725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:09.959008 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "894b64647924425995c6f5aa490b6e45"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "a1a7151d214c50e216dab9aae3531195"
server_key_iv: "5c8e58844d6bcc8ff656e8d525134ea7"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.959547 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "894b64647924425995c6f5aa490b6e45"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "a1a7151d214c50e216dab9aae3531195"
server_key_iv: "5c8e58844d6bcc8ff656e8d525134ea7"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.963285 3725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 20:08:09.965873 3740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:09.967156 3725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:08:09.967257 3725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "894b64647924425995c6f5aa490b6e45"
format_stamp: "Formatted at 2026-09-24 20:08:09 on dist-test-slave-2pm7"
server_key: "a1a7151d214c50e216dab9aae3531195"
server_key_iv: "5c8e58844d6bcc8ff656e8d525134ea7"
server_key_version: "encryptionkey@0"
I20260924 20:08:09.967345 3725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:09.981178 3725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:09.984498 3725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:09.984668 3725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:09.985215 3725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:09.986007 3725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:09.986052 3725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:09.986095 3725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:09.986114 3725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:09.994910 3725 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:38229
I20260924 20:08:09.994935 3853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:38229 every 8 connection(s)
I20260924 20:08:09.995853 3725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:10.001537 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 3725
I20260924 20:08:10.001758 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:10.002118 462 external_mini_cluster.cc:1428] Setting key 8b8d3f370b667ac83cf09380c9793bbf
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[3484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:10.011934 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:09.997713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:60737 (local address 127.0.115.190:44363)
0924 20:08:09.997855 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:09.997859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:09.998720 (+ 861us) server_negotiation.cc:407] Connection header received
0924 20:08:10.000192 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:10.000196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:10.000255 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:10.000336 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:10.001796 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.002692 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.004243 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.004461 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.004545 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:10.007409 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:10.007416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:10.007422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:10.007455 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:10.009449 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:10.010067 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:10.010073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:10.010077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:10.010172 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:10.010611 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:10.010616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:10.010618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:10.010899 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:10.011014 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:10.011565 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 20:08:10.011712 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:08:10.012835 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:09.998105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44363 (local address 127.0.115.130:60737)
0924 20:08:09.998548 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:09.998591 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:09.999920 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:10.000509 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:10.000516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:10.000999 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:10.001576 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:10.001591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.002854 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.002860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:10.004107 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:10.004120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.004638 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.004646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:10.004918 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:10.005597 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:10.005616 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:10.007224 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:10.009639 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:10.009645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:10.009659 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:10.009958 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:10.010355 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:10.010358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:10.010360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:10.010505 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:10.011022 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:10.011027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:10.011295 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 20:08:10.011472 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 20:08:10.011792 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260924 20:08:10.014189 3854 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44363
I20260924 20:08:10.014457 3854 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:10.015002 3854 heartbeater.cc:511] Master 127.0.115.190:44363 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:08:10.016213 3529 ts_manager.cc:194] Registered new tserver with Master: 894b64647924425995c6f5aa490b6e45 (127.0.115.130:38229)
I20260924 20:08:10.016832 3529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:60737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:08:10.031211 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44363
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:10.144865 3859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:10.145202 3859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:10.145285 3859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:10.149488 3859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:10.149582 3859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:10.149679 3859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:10.154932 3859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:10.156105 3859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:10.157194 3859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:10.167488 3865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:10.167520 3864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:10.167599 3859 server_base.cc:1034] running on GCE node
W20260924 20:08:10.167490 3867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:10.168304 3859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:10.169008 3859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:10.170245 3859 hybrid_clock.cc:648] HybridClock initialized: now 1790280490170231 us; error 31 us; skew 500 ppm
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[3484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:10.173345 3859 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:10.174249 3859 webserver.cc:460] Webserver started at http://127.0.115.131:38653/ using document root <none> and password file <none>
I20260924 20:08:10.174857 3859 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:10.174957 3859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:10.175232 3859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:10.177570 3859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "d1af35ccfa374039be321988062659bb"
format_stamp: "Formatted at 2026-09-24 20:08:10 on dist-test-slave-2pm7"
server_key: "4e11dd3cf5339aa76f593ed1290e5e65"
server_key_iv: "111b472ac073c61883f278c8b84fb7ba"
server_key_version: "encryptionkey@0"
I20260924 20:08:10.178125 3859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "d1af35ccfa374039be321988062659bb"
format_stamp: "Formatted at 2026-09-24 20:08:10 on dist-test-slave-2pm7"
server_key: "4e11dd3cf5339aa76f593ed1290e5e65"
server_key_iv: "111b472ac073c61883f278c8b84fb7ba"
server_key_version: "encryptionkey@0"
I20260924 20:08:10.181890 3859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:08:10.184396 3874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:10.185634 3859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:10.185878 3859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "d1af35ccfa374039be321988062659bb"
format_stamp: "Formatted at 2026-09-24 20:08:10 on dist-test-slave-2pm7"
server_key: "4e11dd3cf5339aa76f593ed1290e5e65"
server_key_iv: "111b472ac073c61883f278c8b84fb7ba"
server_key_version: "encryptionkey@0"
I20260924 20:08:10.185992 3859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:10.202648 3859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:10.206104 3859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:10.206318 3859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:10.206929 3859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:10.207772 3859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:10.207844 3859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:10.207919 3859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:10.207995 3859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:10.216609 3859 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:40977
I20260924 20:08:10.216636 3987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:40977 every 8 connection(s)
I20260924 20:08:10.217543 3859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:10.217808 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 3859
I20260924 20:08:10.217935 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:10.218286 462 external_mini_cluster.cc:1428] Setting key 643bf716df19b08d457314fb0324744f
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[3484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:10.231241 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:10.219481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:47825 (local address 127.0.115.190:44363)
0924 20:08:10.219636 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:10.219640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:10.220866 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 20:08:10.222355 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:10.222357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:10.222402 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:10.222471 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:10.223797 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.224421 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.225518 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.225726 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.225801 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:10.228145 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:10.228148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:10.228151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:10.228181 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:10.229409 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:10.229859 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:10.229861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:10.229863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:10.229935 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:10.230213 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:10.230215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:10.230217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:10.230386 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:10.230460 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:10.230945 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 20:08:10.231038 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:08:10.232299 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:10.220074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44363 (local address 127.0.115.131:47825)
0924 20:08:10.220646 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:10.220756 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:10.222169 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:10.222698 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:10.222720 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:10.223193 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:10.223676 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:10.223689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.224527 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.224529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:10.225402 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:10.225412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.225853 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.225856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:10.225993 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:10.226518 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:10.226533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:10.227984 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:10.229509 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:10.229514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:10.229525 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:10.229774 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:10.230012 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:10.230014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:10.230015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:10.230137 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:10.230470 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:10.230474 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:10.230715 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 20:08:10.230912 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 20:08:10.231227 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":144,"threads_started":1}
I20260924 20:08:10.233413 3988 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44363
I20260924 20:08:10.233700 3988 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:10.234185 3988 heartbeater.cc:511] Master 127.0.115.190:44363 requested a full tablet report, sending...
I20260924 20:08:10.235174 3529 ts_manager.cc:194] Registered new tserver with Master: d1af35ccfa374039be321988062659bb (127.0.115.131:40977)
I20260924 20:08:10.235730 3529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:47825
I20260924 20:08:10.244388 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[3484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280490, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[3484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280490, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:10.270349 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:10.261547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.0.115.190:44363)
0924 20:08:10.261720 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:10.261725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:10.261900 (+ 175us) server_negotiation.cc:407] Connection header received
0924 20:08:10.262089 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:10.262092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:10.262138 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:10.262225 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:10.262932 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.263467 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.264372 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:10.264596 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:10.264727 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:10.267080 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:10.267086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:10.267093 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:10.267119 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:10.268636 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:10.269069 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:10.269077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:10.269078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:10.269152 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:10.269438 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:10.269441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:10.269443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:10.269679 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:10.269822 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:10.270054 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 20:08:10.270180 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 20:08:10.270823 3529 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48810
I20260924 20:08:10.271955 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 3591
I20260924 20:08:10.278131 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 3725
I20260924 20:08:10.282985 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 3859
I20260924 20:08:10.287515 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 3497
2026-09-24T20:08:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2143 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[4000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[4000](info): set up 2 sockets
Sep 24 20:08:10 dist-test-slave-2pm7 krb5kdc[4000](info): commencing operation
krb5kdc: starting...
W20260924 20:08:11.340387 462 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:11 dist-test-slave-2pm7 krb5kdc[4000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:11.466980 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:45349
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:45349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:11.584189 4013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:11.584520 4013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:11.584607 4013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:11.588739 4013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:11.588840 4013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:11.588878 4013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:11.588915 4013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:11.588948 4013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:11.594487 4013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:45349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:45349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:11.595650 4013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:11.596771 4013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:11.605602 4019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:11.605752 4013 server_base.cc:1034] running on GCE node
W20260924 20:08:11.605756 4021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:11.605959 4018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:11.606550 4013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:11.607774 4013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:11.609000 4013 hybrid_clock.cc:648] HybridClock initialized: now 1790280491608968 us; error 50 us; skew 500 ppm
Sep 24 20:08:11 dist-test-slave-2pm7 krb5kdc[4000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:11.612196 4013 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:11.613154 4013 webserver.cc:460] Webserver started at http://127.0.115.190:40747/ using document root <none> and password file <none>
I20260924 20:08:11.613791 4013 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:11.613891 4013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:11.614159 4013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:11.616437 4013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "a55a11d2215b41bb95771176fab14f4d"
format_stamp: "Formatted at 2026-09-24 20:08:11 on dist-test-slave-2pm7"
server_key: "908389d17a50a21db199625a39b855a0"
server_key_iv: "7d98226c64ef21b9eda910599d419c9c"
server_key_version: "encryptionkey@0"
I20260924 20:08:11.617020 4013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "a55a11d2215b41bb95771176fab14f4d"
format_stamp: "Formatted at 2026-09-24 20:08:11 on dist-test-slave-2pm7"
server_key: "908389d17a50a21db199625a39b855a0"
server_key_iv: "7d98226c64ef21b9eda910599d419c9c"
server_key_version: "encryptionkey@0"
I20260924 20:08:11.620906 4013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:08:11.623781 4028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:11.625162 4013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:11.625325 4013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "a55a11d2215b41bb95771176fab14f4d"
format_stamp: "Formatted at 2026-09-24 20:08:11 on dist-test-slave-2pm7"
server_key: "908389d17a50a21db199625a39b855a0"
server_key_iv: "7d98226c64ef21b9eda910599d419c9c"
server_key_version: "encryptionkey@0"
I20260924 20:08:11.625473 4013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:11.639518 4013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:11.642544 4013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:11.642761 4013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:11.650350 4013 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:45349
I20260924 20:08:11.650372 4080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:45349 every 8 connection(s)
I20260924 20:08:11.651345 4013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:11.653569 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4013
I20260924 20:08:11.653743 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:11.654044 462 external_mini_cluster.cc:1428] Setting key baa9a3fb507a88379bb3487013927f8a
I20260924 20:08:11.654366 4081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:11.661684 4081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d: Bootstrap starting.
Sep 24 20:08:11 dist-test-slave-2pm7 krb5kdc[4000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:11.664023 4081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:11.664961 4081 log.cc:824] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:11.667372 4081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d: No bootstrap required, opened a new log
I20260924 20:08:11.669600 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:11.655469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.0.115.190:45349)
0924 20:08:11.655908 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:11.655928 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:11.655991 (+ 63us) server_negotiation.cc:407] Connection header received
0924 20:08:11.656818 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:11.656843 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:11.657268 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:11.657646 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:11.658376 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:11.659869 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:11.660797 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:11.661458 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:11.661617 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:11.664284 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:11.664291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:11.664311 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:11.664348 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:11.666220 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:11.666789 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:11.666796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:11.666802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:11.666940 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:11.667245 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:11.667248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:11.667250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:11.667779 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:11.668002 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:11.668315 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 20:08:11.668617 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260924 20:08:11.671842 4081 raft_consensus.cc:348] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a11d2215b41bb95771176fab14f4d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45349 } }
I20260924 20:08:11.672134 4081 raft_consensus.cc:374] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:11.672190 4081 raft_consensus.cc:729] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55a11d2215b41bb95771176fab14f4d, State: Initialized, Role: FOLLOWER
I20260924 20:08:11.672598 4081 consensus_queue.cc:260] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a11d2215b41bb95771176fab14f4d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45349 } }
I20260924 20:08:11.672722 4081 raft_consensus.cc:388] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:11.672768 4081 raft_consensus.cc:482] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:11.672840 4081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:11.673761 4081 raft_consensus.cc:504] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a11d2215b41bb95771176fab14f4d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45349 } }
I20260924 20:08:11.674054 4081 leader_election.cc:302] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a55a11d2215b41bb95771176fab14f4d; no voters:
I20260924 20:08:11.674350 4081 leader_election.cc:288] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:11.674435 4086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:11.674669 4086 raft_consensus.cc:686] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [term 1 LEADER]: Becoming Leader. State: Replica: a55a11d2215b41bb95771176fab14f4d, State: Running, Role: LEADER
I20260924 20:08:11.674999 4086 consensus_queue.cc:237] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a11d2215b41bb95771176fab14f4d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45349 } }
I20260924 20:08:11.675712 4081 sys_catalog.cc:564] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:11.676324 4086 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a55a11d2215b41bb95771176fab14f4d. Latest consensus state: current_term: 1 leader_uuid: "a55a11d2215b41bb95771176fab14f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a11d2215b41bb95771176fab14f4d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45349 } } }
I20260924 20:08:11.676524 4086 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:11.676847 4087 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a55a11d2215b41bb95771176fab14f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a11d2215b41bb95771176fab14f4d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45349 } } }
I20260924 20:08:11.677013 4087 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:11.677037 4093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:11.680440 4093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:11.686024 4093 catalog_manager.cc:1347] Generated new cluster ID: e7fd32ba5f134f1fb50ce8de4ccf3dac
I20260924 20:08:11.686101 4093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:11.693593 4093 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:11.694504 4093 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:11.700664 4093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a55a11d2215b41bb95771176fab14f4d: Generated new TSK 0
I20260924 20:08:11.701375 4093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:08:11.738535 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:45349
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:11.869894 4107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:11.870153 4107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:11.870204 4107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:11.874138 4107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:11.874195 4107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:11.874251 4107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:11.879558 4107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:11.880614 4107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:11.881626 4107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:11.891435 4112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:11.891525 4113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:11.891482 4107 server_base.cc:1034] running on GCE node
W20260924 20:08:11.891490 4115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:11.892143 4107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:11.892827 4107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:11.893989 4107 hybrid_clock.cc:648] HybridClock initialized: now 1790280491893972 us; error 30 us; skew 500 ppm
Sep 24 20:08:11 dist-test-slave-2pm7 krb5kdc[4000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:11.897138 4107 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:08:11.898101 4107 webserver.cc:460] Webserver started at http://127.0.115.129:37187/ using document root <none> and password file <none>
I20260924 20:08:11.898710 4107 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:11.898805 4107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:11.899066 4107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:11.901238 4107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "a526fa5c72734a0e97d5386735a9b51e"
format_stamp: "Formatted at 2026-09-24 20:08:11 on dist-test-slave-2pm7"
server_key: "6e7f5588edcc5d6b7d8f7098a005b13b"
server_key_iv: "0910e77cb72a64cde6715e5fd74a49aa"
server_key_version: "encryptionkey@0"
I20260924 20:08:11.901819 4107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "a526fa5c72734a0e97d5386735a9b51e"
format_stamp: "Formatted at 2026-09-24 20:08:11 on dist-test-slave-2pm7"
server_key: "6e7f5588edcc5d6b7d8f7098a005b13b"
server_key_iv: "0910e77cb72a64cde6715e5fd74a49aa"
server_key_version: "encryptionkey@0"
I20260924 20:08:11.905956 4107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:11.908668 4122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:11.910089 4107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:11.910245 4107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "a526fa5c72734a0e97d5386735a9b51e"
format_stamp: "Formatted at 2026-09-24 20:08:11 on dist-test-slave-2pm7"
server_key: "6e7f5588edcc5d6b7d8f7098a005b13b"
server_key_iv: "0910e77cb72a64cde6715e5fd74a49aa"
server_key_version: "encryptionkey@0"
I20260924 20:08:11.910367 4107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:11.932118 4107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:11.935240 4107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:11.935453 4107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:11.936097 4107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:11.936952 4107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:11.937024 4107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:11.937098 4107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:11.937150 4107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:11.946663 4107 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:41591
I20260924 20:08:11.946700 4235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:41591 every 8 connection(s)
I20260924 20:08:11.947767 4107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:11.956274 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4107
I20260924 20:08:11.956403 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:11.956739 462 external_mini_cluster.cc:1428] Setting key 44557fa2c7e6774157a55ab28a2f9b11
Sep 24 20:08:11 dist-test-slave-2pm7 krb5kdc[4000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:11.963729 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:11.949956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:53705 (local address 127.0.115.190:45349)
0924 20:08:11.950127 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:11.950133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:11.951107 (+ 974us) server_negotiation.cc:407] Connection header received
0924 20:08:11.952741 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:11.952744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:11.952792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:11.952875 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:11.954292 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:11.954821 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:11.956217 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:11.956576 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:11.956703 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:11.959810 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:11.959817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:11.959821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:11.959855 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:11.961320 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:11.961985 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:11.961987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:11.961989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:11.962061 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:11.962463 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:11.962465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:11.962466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:11.962714 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:11.962793 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:11.963481 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 20:08:11.963590 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:08:11.965099 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:11.950398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:45349 (local address 127.0.115.129:53705)
0924 20:08:11.950931 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:11.950991 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:11.952512 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:11.953109 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:11.953117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:11.953514 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:11.954137 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:11.954154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:11.954964 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:11.954967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:11.956075 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:11.956090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:11.956739 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:11.956742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:11.956955 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:11.957689 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:11.957713 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:11.959624 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:11.961441 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:11.961450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:11.961471 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:11.961877 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:11.962152 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:11.962158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:11.962164 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:11.962365 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:11.962835 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:11.962845 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:11.963201 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 20:08:11.963439 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 20:08:11.963857 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":151,"threads_started":1}
I20260924 20:08:11.966459 4236 heartbeater.cc:347] Connected to a master server at 127.0.115.190:45349
I20260924 20:08:11.966730 4236 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:08:11.967290 4236 heartbeater.cc:511] Master 127.0.115.190:45349 requested a full tablet report, sending...
I20260924 20:08:11.968999 4045 ts_manager.cc:194] Registered new tserver with Master: a526fa5c72734a0e97d5386735a9b51e (127.0.115.129:41591)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:08:11.970340 4045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:53705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:08:11.986042 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:45349
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:12.103119 4241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:12.103425 4241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:12.103500 4241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:12.107739 4241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:12.107826 4241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:12.107923 4241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:12.113132 4241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:12.114487 4241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:12.115576 4241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:12.125463 4241 server_base.cc:1034] running on GCE node
W20260924 20:08:12.125429 4247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:12.125362 4246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:12.125355 4249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:12.126034 4241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:12.126698 4241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:12.127884 4241 hybrid_clock.cc:648] HybridClock initialized: now 1790280492127868 us; error 27 us; skew 500 ppm
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:12.130944 4241 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:08:12.131871 4241 webserver.cc:460] Webserver started at http://127.0.115.130:43493/ using document root <none> and password file <none>
I20260924 20:08:12.132447 4241 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:12.132509 4241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:12.132741 4241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:12.134797 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "c1c0cfd1e8e7403c956cc45e07a1a01f"
format_stamp: "Formatted at 2026-09-24 20:08:12 on dist-test-slave-2pm7"
server_key: "0f8cb753e8a22e3d8ed3e088a2c9fe4b"
server_key_iv: "b93b5b26ef4de90e944e3b75fd7b3bf1"
server_key_version: "encryptionkey@0"
I20260924 20:08:12.135351 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "c1c0cfd1e8e7403c956cc45e07a1a01f"
format_stamp: "Formatted at 2026-09-24 20:08:12 on dist-test-slave-2pm7"
server_key: "0f8cb753e8a22e3d8ed3e088a2c9fe4b"
server_key_iv: "b93b5b26ef4de90e944e3b75fd7b3bf1"
server_key_version: "encryptionkey@0"
I20260924 20:08:12.139062 4241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:08:12.141512 4256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:12.142825 4241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:12.142962 4241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "c1c0cfd1e8e7403c956cc45e07a1a01f"
format_stamp: "Formatted at 2026-09-24 20:08:12 on dist-test-slave-2pm7"
server_key: "0f8cb753e8a22e3d8ed3e088a2c9fe4b"
server_key_iv: "b93b5b26ef4de90e944e3b75fd7b3bf1"
server_key_version: "encryptionkey@0"
I20260924 20:08:12.143074 4241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:12.161967 4241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:12.165359 4241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:12.165518 4241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:12.166147 4241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:12.166942 4241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:12.166988 4241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:12.167027 4241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:12.167043 4241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:12.176199 4241 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:37635
I20260924 20:08:12.176265 4369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:37635 every 8 connection(s)
I20260924 20:08:12.177099 4241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:12.182561 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4241
I20260924 20:08:12.182698 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:12.182998 462 external_mini_cluster.cc:1428] Setting key 25a69d79c2880417a4f9caa288e3d461
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:12.191869 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:12.179136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:48981 (local address 127.0.115.190:45349)
0924 20:08:12.179293 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:12.179298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:12.180274 (+ 976us) server_negotiation.cc:407] Connection header received
0924 20:08:12.181700 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:12.181704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:12.181752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:12.181828 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:12.183060 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.183758 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.185133 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.185374 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.185487 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:12.188172 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:12.188175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:12.188178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:12.188212 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:12.189762 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.190302 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.190306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.190308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.190381 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.190739 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.190743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.190744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.190938 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:12.191047 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:12.191572 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 20:08:12.191698 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:08:12.192732 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:12.179581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:45349 (local address 127.0.115.130:48981)
0924 20:08:12.180128 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:12.180173 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:12.181490 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:12.181960 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:12.181966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:12.182345 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:12.182910 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:12.182926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.183913 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.183916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:12.185001 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:12.185010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.185527 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.185531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:12.185703 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:12.186331 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:12.186347 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:12.188004 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:12.189886 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:12.189892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:12.189909 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:12.190186 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:12.190482 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:12.190486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:12.190488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:12.190636 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:12.191048 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:12.191056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:12.191316 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 20:08:12.191504 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 20:08:12.191848 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260924 20:08:12.194065 4370 heartbeater.cc:347] Connected to a master server at 127.0.115.190:45349
I20260924 20:08:12.194314 4370 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:08:12.194862 4370 heartbeater.cc:511] Master 127.0.115.190:45349 requested a full tablet report, sending...
I20260924 20:08:12.196091 4045 ts_manager.cc:194] Registered new tserver with Master: c1c0cfd1e8e7403c956cc45e07a1a01f (127.0.115.130:37635)
I20260924 20:08:12.196813 4045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:48981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:08:12.211897 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:45349
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:12.324412 4375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:12.324719 4375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:12.324801 4375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:12.329059 4375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:12.329138 4375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:12.329231 4375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:12.334647 4375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:45349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:12.335752 4375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:12.336884 4375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:12.346869 4380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:12.346885 4381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:12.346906 4383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:12.347119 4375 server_base.cc:1034] running on GCE node
I20260924 20:08:12.347630 4375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:12.348402 4375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:12.349650 4375 hybrid_clock.cc:648] HybridClock initialized: now 1790280492349576 us; error 39 us; skew 500 ppm
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:12.352557 4375 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:12.353542 4375 webserver.cc:460] Webserver started at http://127.0.115.131:46867/ using document root <none> and password file <none>
I20260924 20:08:12.354166 4375 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:12.354261 4375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:12.354506 4375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:12.356727 4375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "90b743c846db43c29a337f81e59d04c0"
format_stamp: "Formatted at 2026-09-24 20:08:12 on dist-test-slave-2pm7"
server_key: "16c7408981a52f3561296b0293433eb3"
server_key_iv: "9a4037bf6b8a9ff930677d52aa830612"
server_key_version: "encryptionkey@0"
I20260924 20:08:12.357304 4375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "90b743c846db43c29a337f81e59d04c0"
format_stamp: "Formatted at 2026-09-24 20:08:12 on dist-test-slave-2pm7"
server_key: "16c7408981a52f3561296b0293433eb3"
server_key_iv: "9a4037bf6b8a9ff930677d52aa830612"
server_key_version: "encryptionkey@0"
I20260924 20:08:12.361189 4375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 20:08:12.363688 4390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:12.365007 4375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:08:12.365163 4375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "90b743c846db43c29a337f81e59d04c0"
format_stamp: "Formatted at 2026-09-24 20:08:12 on dist-test-slave-2pm7"
server_key: "16c7408981a52f3561296b0293433eb3"
server_key_iv: "9a4037bf6b8a9ff930677d52aa830612"
server_key_version: "encryptionkey@0"
I20260924 20:08:12.365284 4375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:12.380280 4375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:12.383747 4375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:12.384017 4375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:12.384654 4375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:12.385550 4375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:12.385628 4375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:12.385715 4375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:12.385766 4375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:12.394603 4375 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:46741
I20260924 20:08:12.394618 4503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:46741 every 8 connection(s)
I20260924 20:08:12.395537 4375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:12.398819 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4375
I20260924 20:08:12.398983 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:12.399376 462 external_mini_cluster.cc:1428] Setting key 3ced6aa3ab8f051f4b034128b9691499
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:12.410459 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:12.397574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:57255 (local address 127.0.115.190:45349)
0924 20:08:12.397743 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:12.397747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:12.398589 (+ 842us) server_negotiation.cc:407] Connection header received
0924 20:08:12.400541 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:12.400545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:12.400609 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:12.400726 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:12.402124 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.402725 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.404055 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.404308 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.404485 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:12.407109 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:12.407112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:12.407115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:12.407154 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:12.408681 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.409148 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.409151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.409152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.409227 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.409524 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.409527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.409528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.409698 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:12.409774 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:12.410237 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 20:08:12.410337 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:08:12.411629 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:12.397861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:45349 (local address 127.0.115.131:57255)
0924 20:08:12.398360 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:12.398407 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:12.400294 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:12.400921 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:12.400928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:12.401320 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:12.401958 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:12.401974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.402873 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.402882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:12.403856 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:12.403867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.404531 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.404536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:12.404730 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:12.405309 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:12.405327 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:12.406937 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:12.408793 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:12.408797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:12.408810 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:12.409051 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:12.409311 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:12.409313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:12.409315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:12.409443 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:12.409786 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:12.409791 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:12.410035 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 20:08:12.410189 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 20:08:12.410435 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":114,"threads_started":1}
I20260924 20:08:12.412808 4504 heartbeater.cc:347] Connected to a master server at 127.0.115.190:45349
I20260924 20:08:12.413074 4504 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:12.413651 4504 heartbeater.cc:511] Master 127.0.115.190:45349 requested a full tablet report, sending...
I20260924 20:08:12.414806 4044 ts_manager.cc:194] Registered new tserver with Master: 90b743c846db43c29a337f81e59d04c0 (127.0.115.131:46741)
I20260924 20:08:12.415442 4044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:57255
I20260924 20:08:12.425558 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:08:12.451857 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:12.442129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.0.115.129:41591)
0924 20:08:12.442425 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:12.442440 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:12.442466 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:08:12.442520 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:12.442525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:12.442727 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:12.442847 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:12.443504 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.444358 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.445232 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.445604 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.445687 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:12.448540 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:12.448549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:12.448563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:12.448588 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:12.450200 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.450624 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.450629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.450630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.450705 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.450952 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.450955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.450956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.451200 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:12.451373 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:12.451575 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 20:08:12.451701 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:12.463091 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:12.454976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.0.115.190:45349)
0924 20:08:12.455151 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:12.455158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:12.455194 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:08:12.455352 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:12.455355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:12.455397 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:12.455481 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:12.456182 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.456697 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.457473 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:12.457698 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:12.457809 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:12.460230 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:12.460233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:12.460234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:12.460259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:12.461664 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.462017 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.462020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.462021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.462092 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:12.462327 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:12.462329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:12.462330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:12.462544 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:12.462703 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:12.462878 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 20:08:12.462970 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 20:08:12.463521 4044 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44374
I20260924 20:08:12.464217 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4107
I20260924 20:08:12.470259 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4241
I20260924 20:08:12.476159 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4375
I20260924 20:08:12.482507 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4013
2026-09-24T20:08:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2194 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4516](info): set up 2 sockets
Sep 24 20:08:12 dist-test-slave-2pm7 krb5kdc[4516](info): commencing operation
krb5kdc: starting...
W20260924 20:08:13.548324 462 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:13 dist-test-slave-2pm7 krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:13.674535 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44563
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:13.789444 4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:13.789798 4529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:13.789877 4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:13.794000 4529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:13.794060 4529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:13.794080 4529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:13.794096 4529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:13.794111 4529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:13.799492 4529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:13.800731 4529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:13.801826 4529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:13.810865 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:13.810877 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:13.810880 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:13.811293 4529 server_base.cc:1034] running on GCE node
I20260924 20:08:13.811851 4529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:13.813054 4529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:13.814227 4529 hybrid_clock.cc:648] HybridClock initialized: now 1790280493814195 us; error 48 us; skew 500 ppm
Sep 24 20:08:13 dist-test-slave-2pm7 krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:13.817070 4529 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:13.817894 4529 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:13.817991 4529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:13.818239 4529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:13.820443 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "502f73e83bd34e6d9a825500edab7f53"
format_stamp: "Formatted at 2026-09-24 20:08:13 on dist-test-slave-2pm7"
server_key: "b7c4cc6f210dcb4d36ccd66e81a3bb6a"
server_key_iv: "abc962ffa7c0a2f71ebd44599792a994"
server_key_version: "encryptionkey@0"
I20260924 20:08:13.821012 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "502f73e83bd34e6d9a825500edab7f53"
format_stamp: "Formatted at 2026-09-24 20:08:13 on dist-test-slave-2pm7"
server_key: "b7c4cc6f210dcb4d36ccd66e81a3bb6a"
server_key_iv: "abc962ffa7c0a2f71ebd44599792a994"
server_key_version: "encryptionkey@0"
I20260924 20:08:13.824860 4529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:08:13.827522 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:13.828948 4529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:13.829093 4529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "502f73e83bd34e6d9a825500edab7f53"
format_stamp: "Formatted at 2026-09-24 20:08:13 on dist-test-slave-2pm7"
server_key: "b7c4cc6f210dcb4d36ccd66e81a3bb6a"
server_key_iv: "abc962ffa7c0a2f71ebd44599792a994"
server_key_version: "encryptionkey@0"
I20260924 20:08:13.829224 4529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:13.863315 4529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:13.866806 4529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:13.867033 4529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:13.874848 4529 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44563
I20260924 20:08:13.875164 4595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44563 every 8 connection(s)
I20260924 20:08:13.875609 4529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:13.878545 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:13.881567 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4529
I20260924 20:08:13.881781 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:13.882074 462 external_mini_cluster.cc:1428] Setting key 9deee6450b27e1671ce6fc44ab899140
I20260924 20:08:13.885231 4596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53: Bootstrap starting.
I20260924 20:08:13.888029 4596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:13.888840 4596 log.cc:824] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:13.890991 4596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53: No bootstrap required, opened a new log
Sep 24 20:08:13 dist-test-slave-2pm7 krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:13.894853 4596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f73e83bd34e6d9a825500edab7f53" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44563 } }
I20260924 20:08:13.895090 4596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:13.895185 4596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502f73e83bd34e6d9a825500edab7f53, State: Initialized, Role: FOLLOWER
I20260924 20:08:13.895709 4596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f73e83bd34e6d9a825500edab7f53" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44563 } }
I20260924 20:08:13.895900 4596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:13.896000 4596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:13.896140 4596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:13.897331 4596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f73e83bd34e6d9a825500edab7f53" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44563 } }
I20260924 20:08:13.897684 4596 leader_election.cc:302] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502f73e83bd34e6d9a825500edab7f53; no voters:
I20260924 20:08:13.898049 4596 leader_election.cc:288] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:13.898141 4601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:13.898339 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:13.883625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.115.190:44563)
0924 20:08:13.884123 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:13.884146 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:13.884194 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:08:13.885025 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:13.885049 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:13.885490 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:13.885871 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:13.886572 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:13.888112 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:13.888990 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:13.889671 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:13.889856 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:13.892699 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:13.892704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:13.892719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:13.892747 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:13.895022 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:13.895567 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:13.895573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:13.895579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:13.895711 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:13.896036 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:13.896040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:13.896042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:13.896532 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:13.896700 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:13.897009 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 20:08:13.897339 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":107,"threads_started":1}
I20260924 20:08:13.898391 4601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [term 1 LEADER]: Becoming Leader. State: Replica: 502f73e83bd34e6d9a825500edab7f53, State: Running, Role: LEADER
I20260924 20:08:13.898742 4601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f73e83bd34e6d9a825500edab7f53" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44563 } }
I20260924 20:08:13.900101 4601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502f73e83bd34e6d9a825500edab7f53. Latest consensus state: current_term: 1 leader_uuid: "502f73e83bd34e6d9a825500edab7f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f73e83bd34e6d9a825500edab7f53" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44563 } } }
I20260924 20:08:13.900219 4601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:13.900859 4596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:13.901377 4602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502f73e83bd34e6d9a825500edab7f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f73e83bd34e6d9a825500edab7f53" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44563 } } }
I20260924 20:08:13.901535 4602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:13.901788 4605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:13.905174 4605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:13.910983 4605 catalog_manager.cc:1347] Generated new cluster ID: f40ee5804a3f4ba29c883a85066018ec
I20260924 20:08:13.911069 4605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:13.924245 4605 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:13.925299 4605 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:13.931481 4605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53: Generated new TSK 0
I20260924 20:08:13.932358 4605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:08:13.963563 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44563
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:14.078269 4622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:14.078615 4622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:14.078712 4622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:14.082930 4622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:14.083029 4622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:14.083135 4622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:14.088558 4622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:14.089666 4622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:14.090755 4622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:14.100554 4628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:14.100507 4630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:14.100507 4627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:14.100764 4622 server_base.cc:1034] running on GCE node
I20260924 20:08:14.101365 4622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:14.102118 4622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:14.103282 4622 hybrid_clock.cc:648] HybridClock initialized: now 1790280494103265 us; error 26 us; skew 500 ppm
Sep 24 20:08:14 dist-test-slave-2pm7 krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:14.106271 4622 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:08:14.107069 4622 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:14.107167 4622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:14.107410 4622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:14.109586 4622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "5ec8580f99964f88a600d7fb1726f593"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "8fd506ad73874bbefe368c74895252e9"
server_key_iv: "08bbfdd26b694719d80b52914affac14"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.110193 4622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "5ec8580f99964f88a600d7fb1726f593"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "8fd506ad73874bbefe368c74895252e9"
server_key_iv: "08bbfdd26b694719d80b52914affac14"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.114137 4622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 20:08:14.116613 4636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.117865 4622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 20:08:14.118005 4622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "5ec8580f99964f88a600d7fb1726f593"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "8fd506ad73874bbefe368c74895252e9"
server_key_iv: "08bbfdd26b694719d80b52914affac14"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.118124 4622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:14.135037 4622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:14.137989 4622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:14.138253 4622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:14.138877 4622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:14.139731 4622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:14.139803 4622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.139876 4622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:14.139928 4622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.149413 4622 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:33009
I20260924 20:08:14.149434 4749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:33009 every 8 connection(s)
I20260924 20:08:14.150040 4622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:14.150352 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4622
I20260924 20:08:14.150493 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:14.150842 462 external_mini_cluster.cc:1428] Setting key a5ff2c8759ad6194d41ca65ea37878c3
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Sep 24 20:08:14 dist-test-slave-2pm7 krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:14.168107 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.152083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:45661 (local address 127.0.115.190:44563)
0924 20:08:14.152224 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.152231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.153382 (+ 1151us) server_negotiation.cc:407] Connection header received
0924 20:08:14.154991 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.154994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.155049 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.155133 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:14.156418 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.157037 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.158249 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.158489 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.158601 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.163804 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:14.163811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:14.163817 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:14.163854 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:14.165570 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.166252 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.166255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.166256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.166330 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.166745 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.166747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.166748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.166952 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:14.167067 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.167804 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.167923 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:08:14.169459 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.152640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44563 (local address 127.0.115.129:45661)
0924 20:08:14.153194 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.153257 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.154774 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.155337 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.155344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.155703 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:14.156274 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.156290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.157167 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.157171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.158088 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.158097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.158646 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.158650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.158832 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.159438 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:14.159456 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:14.163603 (+ 4147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:14.165703 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:14.165713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:14.165734 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:14.166139 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:14.166425 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:14.166431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:14.166436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:14.166640 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:14.167097 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:14.167107 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:14.167509 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.167743 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.168205 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":116,"threads_started":1}
I20260924 20:08:14.170892 4750 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44563
I20260924 20:08:14.171154 4750 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:14.171645 4750 heartbeater.cc:511] Master 127.0.115.190:44563 requested a full tablet report, sending...
I20260924 20:08:14.173339 4560 ts_manager.cc:194] Registered new tserver with Master: 5ec8580f99964f88a600d7fb1726f593 (127.0.115.129:33009)
I20260924 20:08:14.174548 4560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:45661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:08:14.179544 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44563
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:14.295615 4755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:14.295887 4755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:14.295934 4755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:14.300069 4755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:14.300125 4755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:14.300223 4755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:14.305787 4755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:14.306941 4755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:14.308053 4755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:14.318138 4760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:14.318142 4761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:14.318199 4763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:14.318588 4755 server_base.cc:1034] running on GCE node
I20260924 20:08:14.319108 4755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:14.319801 4755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:14.320997 4755 hybrid_clock.cc:648] HybridClock initialized: now 1790280494320959 us; error 51 us; skew 500 ppm
Sep 24 20:08:14 dist-test-slave-2pm7 krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:14.324071 4755 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:08:14.324862 4755 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:14.324960 4755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:14.325191 4755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:14.327383 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "b2974612e82a48439ed80d05b16ae5da"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "4e96ccb592c6e19d72d6626c1c03dee3"
server_key_iv: "049e7358e289dd7c632b1531bea594df"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.328028 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "b2974612e82a48439ed80d05b16ae5da"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "4e96ccb592c6e19d72d6626c1c03dee3"
server_key_iv: "049e7358e289dd7c632b1531bea594df"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.331840 4755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:08:14.334784 4769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.336319 4755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:14.336467 4755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "b2974612e82a48439ed80d05b16ae5da"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "4e96ccb592c6e19d72d6626c1c03dee3"
server_key_iv: "049e7358e289dd7c632b1531bea594df"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.336594 4755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:14.371229 4755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:14.375005 4755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:14.375222 4755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:14.375856 4755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:14.376803 4755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:14.376888 4755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.376963 4755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:14.377017 4755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.386135 4755 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:46809
I20260924 20:08:14.386152 4882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:46809 every 8 connection(s)
I20260924 20:08:14.386786 4755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:14.396588 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4755
I20260924 20:08:14.396766 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
Sep 24 20:08:14 dist-test-slave-2pm7 krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:14.397125 462 external_mini_cluster.cc:1428] Setting key 64bce69fb8eccbb758fc48463629f4c9
I20260924 20:08:14.402401 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.388757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:46491 (local address 127.0.115.190:44563)
0924 20:08:14.388927 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.388931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.389831 (+ 900us) server_negotiation.cc:407] Connection header received
0924 20:08:14.391234 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.391238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.391297 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.391380 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:14.392635 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.393198 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.394402 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.394651 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.394771 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.397979 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:14.397983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:14.397986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:14.398010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:14.399618 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.400353 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.400356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.400358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.400489 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.400937 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.400942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.400945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.401225 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:14.401331 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.402020 (+ 689us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.402226 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 20:08:14.403434 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.389127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44563 (local address 127.0.115.130:46491)
0924 20:08:14.389662 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.389709 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.390998 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.391564 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.391571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.391964 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:14.392464 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.392478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.393338 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.393346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.394264 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.394274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.394816 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.394821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.394997 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.395645 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:14.395664 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:14.397750 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:14.399777 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:14.399787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:14.399806 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:14.400249 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:14.400610 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:14.400615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:14.400619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:14.400790 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:14.401372 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:14.401379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:14.401709 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.401960 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.402363 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":109,"threads_started":1}
I20260924 20:08:14.404729 4883 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44563
I20260924 20:08:14.405021 4883 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:14.405498 4883 heartbeater.cc:511] Master 127.0.115.190:44563 requested a full tablet report, sending...
I20260924 20:08:14.406751 4560 ts_manager.cc:194] Registered new tserver with Master: b2974612e82a48439ed80d05b16ae5da (127.0.115.130:46809)
I20260924 20:08:14.407398 4560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:46491
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:08:14.425242 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44563
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:14.538980 4888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:14.539242 4888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:14.539287 4888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:14.543273 4888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:14.543334 4888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:14.543394 4888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:14.548745 4888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:14.549786 4888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:14.550858 4888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:14.560333 4894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:14.560375 4896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:14.560375 4893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:14.560536 4888 server_base.cc:1034] running on GCE node
I20260924 20:08:14.561036 4888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:14.561628 4888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:14.562824 4888 hybrid_clock.cc:648] HybridClock initialized: now 1790280494562808 us; error 26 us; skew 500 ppm
Sep 24 20:08:14 dist-test-slave-2pm7 krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:14.565671 4888 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:14.566339 4888 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:14.566397 4888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:14.566553 4888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:14.568629 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "c4c90c82fa9f4973ae5a52482be07f9d"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "ba68fbcffa42cd9e5c0154010fe0a4a9"
server_key_iv: "7c9671f275513b2c94de39f1a9c9d1f9"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.569131 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "c4c90c82fa9f4973ae5a52482be07f9d"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "ba68fbcffa42cd9e5c0154010fe0a4a9"
server_key_iv: "7c9671f275513b2c94de39f1a9c9d1f9"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.572886 4888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:08:14.575297 4902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.576689 4888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:14.576830 4888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "c4c90c82fa9f4973ae5a52482be07f9d"
format_stamp: "Formatted at 2026-09-24 20:08:14 on dist-test-slave-2pm7"
server_key: "ba68fbcffa42cd9e5c0154010fe0a4a9"
server_key_iv: "7c9671f275513b2c94de39f1a9c9d1f9"
server_key_version: "encryptionkey@0"
I20260924 20:08:14.576941 4888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:14.595739 4888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:14.599152 4888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:14.599359 4888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:14.600028 4888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:14.600950 4888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:14.600996 4888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.601069 4888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:14.601109 4888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:14.610133 4888 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:43891
I20260924 20:08:14.610205 5015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:43891 every 8 connection(s)
I20260924 20:08:14.610788 4888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:14.611714 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 4888
I20260924 20:08:14.611860 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:14.612185 462 external_mini_cluster.cc:1428] Setting key 9042d1e5d068e7b4762b7e2b25ca8e83
Sep 24 20:08:14 dist-test-slave-2pm7 krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:14.625454 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.614418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:48265 (local address 127.0.115.190:44563)
0924 20:08:14.614548 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.614551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.614564 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:14.615603 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.615605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.615646 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.615723 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:14.617087 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.617615 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.618873 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.619088 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.619163 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.621955 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:14.621958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:14.621960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:14.621984 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:14.623314 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.623832 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.623835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.623836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.623908 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.624287 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.624290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.624291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.624483 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:14.624592 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.625166 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.625287 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:08:14.626439 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.613392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44563 (local address 127.0.115.131:48265)
0924 20:08:14.613908 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.613962 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.615241 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.615875 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.615882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.616360 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:14.616949 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.616964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.617739 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.617742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.618763 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.618773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.619227 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.619231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.619381 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.620156 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:14.620174 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:14.621803 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:14.623425 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:14.623430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:14.623444 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:14.623745 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:14.624051 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:14.624054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:14.624055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:14.624199 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:14.624593 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:14.624601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:14.624884 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.625112 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.625439 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":155,"threads_started":1}
I20260924 20:08:14.627503 5016 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44563
I20260924 20:08:14.627724 5016 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:14.628234 5016 heartbeater.cc:511] Master 127.0.115.190:44563 requested a full tablet report, sending...
I20260924 20:08:14.629263 4560 ts_manager.cc:194] Registered new tserver with Master: c4c90c82fa9f4973ae5a52482be07f9d (127.0.115.131:43891)
I20260924 20:08:14.629915 4560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:48265
I20260924 20:08:14.637899 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:08:14.646836 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.639790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.0.115.190:44563)
0924 20:08:14.639910 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.639913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.640167 (+ 254us) server_negotiation.cc:407] Connection header received
0924 20:08:14.640253 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.640255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.640293 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.640356 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:14.641093 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.641618 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.642393 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.642595 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.642669 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.644049 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:14.644052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:14.644054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:14.644075 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:14.645380 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.645811 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.645814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.645815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.645885 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:14.646127 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:14.646129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:14.646130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:14.646340 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:14.646443 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.646630 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.646721 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:08:14.649703 4560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:14.652772 4560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:14.667517 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.661419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:43891 (local address 127.0.0.1:46824)
0924 20:08:14.662353 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.662405 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.662534 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.663508 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.663511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.663549 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:14.663659 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.663666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.665105 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.665109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.666158 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.666168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.666978 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.666982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.667089 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.667126 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.667238 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.667364 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":32,"thread_start_us":83,"threads_started":1}
I20260924 20:08:14.667614 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.661578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.0.115.131:43891)
0924 20:08:14.662872 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.662886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.662909 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:08:14.662985 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.662991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.663190 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.663342 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:14.663794 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.664918 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.666292 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.666848 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.666979 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.667143 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.667372 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.667466 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":98,"threads_started":1}
I20260924 20:08:14.667688 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.661446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:33009 (local address 127.0.0.1:58650)
0924 20:08:14.662516 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.662529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.662619 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.663168 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.663174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.663196 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:14.663348 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.663355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.665029 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.665033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.666186 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.666194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.667216 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.667220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.667375 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.667398 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.667463 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.667538 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":52,"thread_start_us":96,"threads_started":1}
I20260924 20:08:14.668059 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.662266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.0.115.129:33009)
0924 20:08:14.662657 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.662667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.662692 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:08:14.662772 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.662777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.662997 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.663182 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:14.663483 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.664768 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.666322 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.667073 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.667332 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.667472 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.667522 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.667609 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":107,"threads_started":1}
I20260924 20:08:14.668531 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.662101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.115.130:46809)
0924 20:08:14.662982 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.662993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.663011 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:08:14.663073 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.663077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.663217 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.663342 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:14.663584 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.664452 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.666130 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.666984 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.667096 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.667256 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.668024 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.668379 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":149,"threads_started":1}
I20260924 20:08:14.668825 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.661720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:46809 (local address 127.0.0.1:35110)
0924 20:08:14.662780 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.662792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.662864 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.663349 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.663352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.663367 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:14.663471 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.663477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.664587 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.664591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.665994 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.666005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.667225 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.667232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.667774 (+ 542us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.667797 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.667868 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.667934 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":428,"thread_start_us":79,"threads_started":1}
I20260924 20:08:14.670805 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 723762915e4f4a4c8384bc93a832e5ef (DEFAULT_TABLE table=test-table [id=486eab244dbc4a428bab866592a4fbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:14.670873 4950 tablet_service.cc:1467] Processing CreateTablet for tablet 723762915e4f4a4c8384bc93a832e5ef (DEFAULT_TABLE table=test-table [id=486eab244dbc4a428bab866592a4fbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:14.671245 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 723762915e4f4a4c8384bc93a832e5ef (DEFAULT_TABLE table=test-table [id=486eab244dbc4a428bab866592a4fbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:14.671851 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723762915e4f4a4c8384bc93a832e5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:14.671917 4950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723762915e4f4a4c8384bc93a832e5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:14.672207 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723762915e4f4a4c8384bc93a832e5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:14.677902 5033 tablet_bootstrap.cc:492] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: Bootstrap starting.
I20260924 20:08:14.679333 5034 tablet_bootstrap.cc:492] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Bootstrap starting.
I20260924 20:08:14.680282 5035 tablet_bootstrap.cc:492] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: Bootstrap starting.
I20260924 20:08:14.680464 5033 tablet_bootstrap.cc:654] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:14.681496 5034 tablet_bootstrap.cc:654] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:14.681519 5033 log.cc:824] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:14.682736 5035 tablet_bootstrap.cc:654] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:14.682809 5034 log.cc:824] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:14.683737 5033 tablet_bootstrap.cc:492] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: No bootstrap required, opened a new log
I20260924 20:08:14.683964 5033 ts_tablet_manager.cc:1397] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260924 20:08:14.684233 5035 log.cc:824] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:14.685007 5034 tablet_bootstrap.cc:492] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: No bootstrap required, opened a new log
I20260924 20:08:14.685197 5034 ts_tablet_manager.cc:1397] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:08:14.686002 5035 tablet_bootstrap.cc:492] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: No bootstrap required, opened a new log
I20260924 20:08:14.686213 5035 ts_tablet_manager.cc:1397] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:08:14.688452 5033 raft_consensus.cc:348] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.688746 5033 raft_consensus.cc:374] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:14.688805 5033 raft_consensus.cc:729] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec8580f99964f88a600d7fb1726f593, State: Initialized, Role: FOLLOWER
I20260924 20:08:14.689080 5034 raft_consensus.cc:348] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.689252 5033 consensus_queue.cc:260] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.689347 5034 raft_consensus.cc:374] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:14.689404 5034 raft_consensus.cc:729] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c90c82fa9f4973ae5a52482be07f9d, State: Initialized, Role: FOLLOWER
I20260924 20:08:14.689877 5034 consensus_queue.cc:260] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.689988 4750 heartbeater.cc:503] Master 127.0.115.190:44563 was elected leader, sending a full tablet report...
I20260924 20:08:14.689924 5035 raft_consensus.cc:348] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.690198 5035 raft_consensus.cc:374] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:14.690250 5035 raft_consensus.cc:729] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2974612e82a48439ed80d05b16ae5da, State: Initialized, Role: FOLLOWER
I20260924 20:08:14.690306 5033 ts_tablet_manager.cc:1428] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260924 20:08:14.690826 5034 ts_tablet_manager.cc:1428] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:08:14.690831 5016 heartbeater.cc:503] Master 127.0.115.190:44563 was elected leader, sending a full tablet report...
I20260924 20:08:14.691743 5035 consensus_queue.cc:260] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.692471 4883 heartbeater.cc:503] Master 127.0.115.190:44563 was elected leader, sending a full tablet report...
I20260924 20:08:14.692764 5035 ts_tablet_manager.cc:1428] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 20:08:14.862346 5017 tablet.cc:2369] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:14.864318 5040 raft_consensus.cc:482] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:14.864509 5040 raft_consensus.cc:504] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.865708 5040 leader_election.cc:288] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec8580f99964f88a600d7fb1726f593 (127.0.115.129:33009), b2974612e82a48439ed80d05b16ae5da (127.0.115.130:46809)
I20260924 20:08:14.870214 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.866105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:46809 (local address 127.0.115.131:44455)
0924 20:08:14.866216 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.866233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.866331 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.866780 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.866783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.866805 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:14.867019 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.867027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.867958 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.867962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.868950 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.869011 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.869823 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.869827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.869931 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.869955 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.870043 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.870105 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"spinlock_wait_cycles":38528}
I20260924 20:08:14.870291 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.866353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:44455 (local address 127.0.115.130:46809)
0924 20:08:14.866496 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.866499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.866511 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:08:14.866591 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.866597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.866651 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.866738 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:14.867179 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.867817 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.869151 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.869690 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.869823 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.869897 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.870085 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.870161 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:08:14.870704 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.866296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:60557 (local address 127.0.115.129:33009)
0924 20:08:14.866450 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.866455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.866875 (+ 420us) server_negotiation.cc:407] Connection header received
0924 20:08:14.867012 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.867015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.867073 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.867157 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:14.867450 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.868025 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.869090 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.869522 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.869619 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.869662 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.870416 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.870510 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:08:14.871040 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.866105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:33009 (local address 127.0.115.131:60557)
0924 20:08:14.866765 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:14.866779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:14.866898 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:14.867192 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:14.867195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:14.867213 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:14.867338 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.867345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.868134 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.868137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.868903 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:14.868911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.870154 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.870158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:14.870255 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.870277 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:08:14.870865 (+ 588us) client_negotiation.cc:240] Negotiation successful
0924 20:08:14.870937 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":277,"spinlock_wait_cycles":768,"thread_start_us":140,"threads_started":1}
I20260924 20:08:14.870990 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723762915e4f4a4c8384bc93a832e5ef" candidate_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2974612e82a48439ed80d05b16ae5da" is_pre_election: true
I20260924 20:08:14.871342 4837 raft_consensus.cc:2454] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c90c82fa9f4973ae5a52482be07f9d in term 0.
I20260924 20:08:14.871699 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723762915e4f4a4c8384bc93a832e5ef" candidate_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec8580f99964f88a600d7fb1726f593" is_pre_election: true
I20260924 20:08:14.872010 4905 leader_election.cc:302] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2974612e82a48439ed80d05b16ae5da, c4c90c82fa9f4973ae5a52482be07f9d; no voters:
I20260924 20:08:14.872063 4704 raft_consensus.cc:2454] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c90c82fa9f4973ae5a52482be07f9d in term 0.
I20260924 20:08:14.872313 5040 raft_consensus.cc:2781] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:14.872426 5040 raft_consensus.cc:482] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:14.872515 5040 raft_consensus.cc:3037] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 0 FOLLOWER]: Advancing to term 1
W20260924 20:08:14.887848 4884 tablet.cc:2369] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:08:14.901499 4751 tablet.cc:2369] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:14.934651 5040 raft_consensus.cc:504] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.935256 5040 leader_election.cc:288] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [CANDIDATE]: Term 1 election: Requested vote from peers 5ec8580f99964f88a600d7fb1726f593 (127.0.115.129:33009), b2974612e82a48439ed80d05b16ae5da (127.0.115.130:46809)
I20260924 20:08:14.935783 4837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723762915e4f4a4c8384bc93a832e5ef" candidate_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2974612e82a48439ed80d05b16ae5da"
I20260924 20:08:14.935834 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723762915e4f4a4c8384bc93a832e5ef" candidate_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec8580f99964f88a600d7fb1726f593"
I20260924 20:08:14.935950 4837 raft_consensus.cc:3037] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:14.936049 4704 raft_consensus.cc:3037] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:14.937179 4837 raft_consensus.cc:2454] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c90c82fa9f4973ae5a52482be07f9d in term 1.
I20260924 20:08:14.937621 4905 leader_election.cc:302] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2974612e82a48439ed80d05b16ae5da, c4c90c82fa9f4973ae5a52482be07f9d; no voters:
I20260924 20:08:14.937834 5040 raft_consensus.cc:2781] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:14.937891 4704 raft_consensus.cc:2454] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c90c82fa9f4973ae5a52482be07f9d in term 1.
I20260924 20:08:14.938107 5040 raft_consensus.cc:686] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 1 LEADER]: Becoming Leader. State: Replica: c4c90c82fa9f4973ae5a52482be07f9d, State: Running, Role: LEADER
I20260924 20:08:14.938777 5040 consensus_queue.cc:237] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } }
I20260924 20:08:14.942330 4560 catalog_manager.cc:5495] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to c4c90c82fa9f4973ae5a52482be07f9d (127.0.115.131). New cstate: current_term: 1 leader_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4c90c82fa9f4973ae5a52482be07f9d" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:14.965389 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:14.961515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.0.115.131:43891)
0924 20:08:14.961664 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:14.961668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:14.961682 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:14.961848 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:14.961850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:14.961906 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:14.961995 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:14.962296 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.962834 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.963688 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:14.963910 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:14.964064 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:14.964414 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:14.964952 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:14.965074 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:14.965142 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 20:08:14.965200 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:08:14.971623 4837 raft_consensus.cc:1261] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 1 FOLLOWER]: Refusing update from remote peer c4c90c82fa9f4973ae5a52482be07f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:14.971630 4704 raft_consensus.cc:1261] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 1 FOLLOWER]: Refusing update from remote peer c4c90c82fa9f4973ae5a52482be07f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:14.972410 5043 consensus_queue.cc:1035] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2974612e82a48439ed80d05b16ae5da" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:14.972647 5040 consensus_queue.cc:1035] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec8580f99964f88a600d7fb1726f593" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:14.982996 5047 mvcc.cc:205] Tried to move back new op lower bound from 7332988907396374528 to 7332988907272536064. Current Snapshot: MvccSnapshot[applied={T|T < 7332988907396374528}]
I20260924 20:08:14.988555 4559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:14.988730 4559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:14.991060 4559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 502f73e83bd34e6d9a825500edab7f53: Sending DeleteTablet for 3 replicas of tablet 723762915e4f4a4c8384bc93a832e5ef
I20260924 20:08:14.991919 4950 tablet_service.cc:1514] Processing DeleteTablet for tablet 723762915e4f4a4c8384bc93a832e5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:14 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:46824
I20260924 20:08:14.991925 4684 tablet_service.cc:1514] Processing DeleteTablet for tablet 723762915e4f4a4c8384bc93a832e5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:14 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:58650
I20260924 20:08:14.992273 4817 tablet_service.cc:1514] Processing DeleteTablet for tablet 723762915e4f4a4c8384bc93a832e5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:14 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:35110
I20260924 20:08:14.992487 5055 tablet_replica.cc:330] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593: stopping tablet replica
I20260924 20:08:14.992868 5055 raft_consensus.cc:2229] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:14.992987 5056 tablet_replica.cc:330] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: stopping tablet replica
I20260924 20:08:14.993211 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4622
I20260924 20:08:14.993211 5055 raft_consensus.cc:2258] T 723762915e4f4a4c8384bc93a832e5ef P 5ec8580f99964f88a600d7fb1726f593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:14.993954 5056 raft_consensus.cc:2229] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:14.994365 5056 raft_consensus.cc:2258] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:14.995553 5057 tablet_replica.cc:330] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: stopping tablet replica
I20260924 20:08:14.995851 5057 raft_consensus.cc:2229] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:14.996205 5057 raft_consensus.cc:2258] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:14.996310 5056 ts_tablet_manager.cc:1905] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:14.999563 5057 ts_tablet_manager.cc:1905] T 723762915e4f4a4c8384bc93a832e5ef P b2974612e82a48439ed80d05b16ae5da: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:08:15.003885 4544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:33009 (error 108)
I20260924 20:08:15.004853 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4755
I20260924 20:08:15.005134 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.004762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:33009 (local address 127.0.0.1:58662)
0924 20:08:15.004933 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.005016 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 20:08:15.007298 4544 catalog_manager.cc:4584] TS 5ec8580f99964f88a600d7fb1726f593 (127.0.115.129:33009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 723762915e4f4a4c8384bc93a832e5ef: Network error: Client connection negotiation failed: client connection to 127.0.115.129:33009: connect: Connection refused (error 111)
I20260924 20:08:15.024535 5056 ts_tablet_manager.cc:1918] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:15.024693 5056 log.cc:1196] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/723762915e4f4a4c8384bc93a832e5ef
I20260924 20:08:15.025050 5056 ts_tablet_manager.cc:1939] T 723762915e4f4a4c8384bc93a832e5ef P c4c90c82fa9f4973ae5a52482be07f9d: Deleting consensus metadata
I20260924 20:08:15.029487 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4888
I20260924 20:08:15.029685 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.029346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:46809 (local address 127.0.0.1:35116)
0924 20:08:15.029523 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.029587 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.130:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 20:08:15.050135 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 4529
2026-09-24T20:08:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2569 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T20:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:15Z Disabled control of system clock
I20260924 20:08:15.082653 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44775
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:08:15.196514 5062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:15.196785 5062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:15.196822 5062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:15.200831 5062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:15.200888 5062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:15.200904 5062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:15.200923 5062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:15.200938 5062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:15.205690 5062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:15.206816 5062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:15.208305 5062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:15.217079 5068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:15.217060 5062 server_base.cc:1034] running on GCE node
W20260924 20:08:15.217065 5067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:15.217044 5070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:15.217856 5062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:15.219096 5062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:15.220291 5062 hybrid_clock.cc:648] HybridClock initialized: now 1790280495220273 us; error 34 us; skew 500 ppm
I20260924 20:08:15.222247 5062 webserver.cc:460] Webserver started at http://127.0.115.190:37939/ using document root <none> and password file <none>
I20260924 20:08:15.222836 5062 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:15.222931 5062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:15.223176 5062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:15.225354 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "523d74c6619e4dc1bef93f833bd7d2a8"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "8977d3269e80592b0ddfa0dbb3fa5932"
server_key_iv: "1a8324629cbef0297577f785e89d7c74"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.225926 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "523d74c6619e4dc1bef93f833bd7d2a8"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "8977d3269e80592b0ddfa0dbb3fa5932"
server_key_iv: "1a8324629cbef0297577f785e89d7c74"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.229779 5062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:15.232257 5076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.233506 5062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:08:15.233670 5062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "523d74c6619e4dc1bef93f833bd7d2a8"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "8977d3269e80592b0ddfa0dbb3fa5932"
server_key_iv: "1a8324629cbef0297577f785e89d7c74"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.233786 5062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:15.238495 5062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:15.239117 5062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:15.239300 5062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:15.246276 5062 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44775
I20260924 20:08:15.246306 5128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44775 every 8 connection(s)
I20260924 20:08:15.247237 5062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:15.248819 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5062
I20260924 20:08:15.248911 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:15.249243 462 external_mini_cluster.cc:1428] Setting key a35df90cb4aa730127f58af199d07318
I20260924 20:08:15.250828 5129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:15.254974 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.250723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.0.115.190:44775)
0924 20:08:15.251374 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.251387 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.251424 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:08:15.252145 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.252164 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.252506 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.252797 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.253679 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.253684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.253696 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.253707 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.253763 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.253779 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.253909 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.254016 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.254284 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":138,"threads_started":1}
I20260924 20:08:15.257347 5129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8: Bootstrap starting.
I20260924 20:08:15.259429 5129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:15.260231 5129 log.cc:824] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:15.262210 5129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8: No bootstrap required, opened a new log
I20260924 20:08:15.265868 5129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44775 } }
I20260924 20:08:15.266080 5129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:15.266158 5129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523d74c6619e4dc1bef93f833bd7d2a8, State: Initialized, Role: FOLLOWER
I20260924 20:08:15.266613 5129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44775 } }
I20260924 20:08:15.266757 5129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:15.266829 5129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:15.266928 5129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:15.267823 5129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44775 } }
I20260924 20:08:15.268193 5129 leader_election.cc:302] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523d74c6619e4dc1bef93f833bd7d2a8; no voters:
I20260924 20:08:15.268543 5129 leader_election.cc:288] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:15.268764 5134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:15.268965 5134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [term 1 LEADER]: Becoming Leader. State: Replica: 523d74c6619e4dc1bef93f833bd7d2a8, State: Running, Role: LEADER
I20260924 20:08:15.269277 5134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44775 } }
I20260924 20:08:15.269814 5129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:15.270880 5136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523d74c6619e4dc1bef93f833bd7d2a8. Latest consensus state: current_term: 1 leader_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44775 } } }
I20260924 20:08:15.271013 5136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:15.271414 5143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:15.271381 5135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523d74c6619e4dc1bef93f833bd7d2a8" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44775 } } }
I20260924 20:08:15.272132 5135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:15.274750 5143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:15.280610 5143 catalog_manager.cc:1347] Generated new cluster ID: 073420591ba24fb6bb0cbe18c8bdd72a
I20260924 20:08:15.280679 5143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:15.290839 5143 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:15.291162 5143 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:15.306102 5143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8: Generated new TSK 0
I20260924 20:08:15.306880 5143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:08:15.321172 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44775
--builtin_ntp_servers=127.0.115.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:08:15.439284 5153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:15.439559 5153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:15.439632 5153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:15.443708 5153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:15.443768 5153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:15.443854 5153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:15.448792 5153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:15.449918 5153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:15.451357 5153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:15.461478 5161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:15.461684 5153 server_base.cc:1034] running on GCE node
W20260924 20:08:15.461485 5158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:15.461491 5159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:15.462288 5153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:15.462942 5153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:15.464135 5153 hybrid_clock.cc:648] HybridClock initialized: now 1790280495464118 us; error 26 us; skew 500 ppm
I20260924 20:08:15.466012 5153 webserver.cc:460] Webserver started at http://127.0.115.129:35119/ using document root <none> and password file <none>
I20260924 20:08:15.466615 5153 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:15.466707 5153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:15.466960 5153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:15.469177 5153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "fb67a11d4d584eb590124b5e5bfd06a7"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "d00003f85fa6db96eda652a012492517"
server_key_iv: "8a8a8bea721f7e40f78361b97601a458"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.469772 5153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "fb67a11d4d584eb590124b5e5bfd06a7"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "d00003f85fa6db96eda652a012492517"
server_key_iv: "8a8a8bea721f7e40f78361b97601a458"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.473912 5153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:15.476610 5167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.477862 5153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:15.478009 5153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "fb67a11d4d584eb590124b5e5bfd06a7"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "d00003f85fa6db96eda652a012492517"
server_key_iv: "8a8a8bea721f7e40f78361b97601a458"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.478147 5153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:15.483379 5153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:15.484131 5153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:15.484326 5153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:15.485112 5153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:15.486506 5153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:15.486569 5153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.486639 5153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:15.486675 5153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.496039 5153 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:39069
I20260924 20:08:15.496042 5280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:39069 every 8 connection(s)
I20260924 20:08:15.496971 5153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:15.498029 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5153
I20260924 20:08:15.498159 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:15.498483 462 external_mini_cluster.cc:1428] Setting key fa2a29d2758cf1bcc78c788a38630f3d
I20260924 20:08:15.500622 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44775
--builtin_ntp_servers=127.0.115.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 20:08:15.504042 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.499034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:58853 (local address 127.0.115.190:44775)
0924 20:08:15.500085 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.500092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.500689 (+ 597us) server_negotiation.cc:407] Connection header received
0924 20:08:15.502162 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.502165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.502199 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.502267 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.503533 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.503536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.503538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.503549 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.503569 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.503577 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.503667 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.503841 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.503879 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953}
I20260924 20:08:15.505227 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.499558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44775 (local address 127.0.115.129:58853)
0924 20:08:15.500489 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.500547 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:15.501902 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:15.502453 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:15.502464 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:15.502584 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:15.503226 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:15.503230 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:15.503285 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:15.503375 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:15.503683 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:15.503694 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:15.503702 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:08:15.503877 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 20:08:15.504147 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":145,"threads_started":1}
I20260924 20:08:15.506553 5281 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44775
I20260924 20:08:15.506808 5281 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:15.507246 5281 heartbeater.cc:511] Master 127.0.115.190:44775 requested a full tablet report, sending...
I20260924 20:08:15.508879 5093 ts_manager.cc:194] Registered new tserver with Master: fb67a11d4d584eb590124b5e5bfd06a7 (127.0.115.129:39069)
W20260924 20:08:15.612496 5284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:15.612808 5284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:15.612880 5284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:15.617084 5284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:15.617167 5284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:15.617260 5284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:15.622505 5284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:15.623661 5284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:15.625159 5284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:15.635201 5292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:15.635270 5290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:15.635397 5284 server_base.cc:1034] running on GCE node
W20260924 20:08:15.635268 5289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:15.635850 5284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:15.636584 5284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:15.637773 5284 hybrid_clock.cc:648] HybridClock initialized: now 1790280495637753 us; error 30 us; skew 500 ppm
I20260924 20:08:15.639611 5284 webserver.cc:460] Webserver started at http://127.0.115.130:37427/ using document root <none> and password file <none>
I20260924 20:08:15.640295 5284 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:15.640386 5284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:15.640626 5284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:15.642719 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "064372eeea2c45aeba02671708c583d6"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "e69e51bbd4b14e09246a1ad81aca71fc"
server_key_iv: "8ae5b64fc3b6a59c7ee3cf09b1d6f93b"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.643301 5284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "064372eeea2c45aeba02671708c583d6"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "e69e51bbd4b14e09246a1ad81aca71fc"
server_key_iv: "8ae5b64fc3b6a59c7ee3cf09b1d6f93b"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.647055 5284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260924 20:08:15.649571 5298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.650759 5284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:15.650892 5284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "064372eeea2c45aeba02671708c583d6"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "e69e51bbd4b14e09246a1ad81aca71fc"
server_key_iv: "8ae5b64fc3b6a59c7ee3cf09b1d6f93b"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.651005 5284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:15.655575 5284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:15.656189 5284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:15.656378 5284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:15.656968 5284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:15.657998 5284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:15.658046 5284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.658083 5284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:15.658099 5284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.666987 5284 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:45291
I20260924 20:08:15.667047 5411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:45291 every 8 connection(s)
I20260924 20:08:15.667879 5284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:15.674186 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.669805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35729 (local address 127.0.115.190:44775)
0924 20:08:15.669945 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.669948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.671088 (+ 1140us) server_negotiation.cc:407] Connection header received
0924 20:08:15.672510 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.672512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.672551 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.672627 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.673809 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.673812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.673813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.673823 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.673838 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.673845 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.673916 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.674042 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.674079 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:08:15.675086 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.670143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44775 (local address 127.0.115.130:35729)
0924 20:08:15.670933 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.670973 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:15.672314 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:15.672842 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:15.672849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:15.672934 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:15.673562 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:15.673564 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:15.673611 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:15.673678 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:15.673918 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:15.673925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:15.673929 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:08:15.674064 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 20:08:15.674282 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260924 20:08:15.676309 5412 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44775
I20260924 20:08:15.676548 5412 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:15.676999 5412 heartbeater.cc:511] Master 127.0.115.190:44775 requested a full tablet report, sending...
I20260924 20:08:15.677546 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5284
I20260924 20:08:15.677691 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:15.678011 462 external_mini_cluster.cc:1428] Setting key ccb47b91fe9b64230e4030f230e05bd6
I20260924 20:08:15.678195 5093 ts_manager.cc:194] Registered new tserver with Master: 064372eeea2c45aeba02671708c583d6 (127.0.115.130:45291)
I20260924 20:08:15.680004 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44775
--builtin_ntp_servers=127.0.115.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:08:15.792546 5415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:15.792831 5415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:15.792873 5415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:15.797015 5415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:15.797070 5415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:15.797155 5415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:15.802029 5415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:15.803153 5415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:15.804644 5415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:15.814867 5421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:15.814858 5423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:15.814883 5420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:15.815074 5415 server_base.cc:1034] running on GCE node
I20260924 20:08:15.815543 5415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:15.816285 5415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:15.817498 5415 hybrid_clock.cc:648] HybridClock initialized: now 1790280495817458 us; error 52 us; skew 500 ppm
I20260924 20:08:15.819442 5415 webserver.cc:460] Webserver started at http://127.0.115.131:37751/ using document root <none> and password file <none>
I20260924 20:08:15.820159 5415 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:15.820258 5415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:15.820497 5415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:15.822675 5415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "8236f6c9dbce4ea08bfb896b0946a5a5"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "2e00d8250e472936171bce044fd5c112"
server_key_iv: "3dd9d574427f5d1fcf2b5b61f8096d56"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.823274 5415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "8236f6c9dbce4ea08bfb896b0946a5a5"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "2e00d8250e472936171bce044fd5c112"
server_key_iv: "3dd9d574427f5d1fcf2b5b61f8096d56"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.827194 5415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:08:15.829700 5429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.831032 5415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:15.831171 5415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "8236f6c9dbce4ea08bfb896b0946a5a5"
format_stamp: "Formatted at 2026-09-24 20:08:15 on dist-test-slave-2pm7"
server_key: "2e00d8250e472936171bce044fd5c112"
server_key_iv: "3dd9d574427f5d1fcf2b5b61f8096d56"
server_key_version: "encryptionkey@0"
I20260924 20:08:15.831287 5415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:15.835932 5415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:15.836560 5415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:15.836800 5415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:15.837316 5415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:15.838253 5415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:15.838297 5415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.838335 5415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:15.838349 5415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:15.846861 5415 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44247
I20260924 20:08:15.846884 5542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44247 every 8 connection(s)
I20260924 20:08:15.847886 5415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:15.854270 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.849894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:60339 (local address 127.0.115.190:44775)
0924 20:08:15.850031 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.850035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.851075 (+ 1040us) server_negotiation.cc:407] Connection header received
0924 20:08:15.852505 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.852509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.852555 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.852666 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.853787 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.853790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.853793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.853806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.853836 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.853845 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.853946 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.854082 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.854129 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:08:15.855149 5545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.850245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44775 (local address 127.0.115.131:60339)
0924 20:08:15.850918 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.850959 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:15.852292 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:15.852840 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:15.852847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:15.852935 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:15.853520 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:15.853521 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:15.853572 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:15.853640 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:15.853929 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:15.853937 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:15.853942 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:08:15.854127 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 20:08:15.854375 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260924 20:08:15.856308 5543 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44775
I20260924 20:08:15.856385 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5415
I20260924 20:08:15.856493 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:15.856563 5543 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:15.856863 462 external_mini_cluster.cc:1428] Setting key 042af20f246d031c3d31e42e65ffeb38
I20260924 20:08:15.857029 5543 heartbeater.cc:511] Master 127.0.115.190:44775 requested a full tablet report, sending...
I20260924 20:08:15.858292 5093 ts_manager.cc:194] Registered new tserver with Master: 8236f6c9dbce4ea08bfb896b0946a5a5 (127.0.115.131:44247)
I20260924 20:08:15.859046 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:08:15.862300 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.860932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.0.115.190:44775)
0924 20:08:15.861071 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.861075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.861237 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:08:15.861334 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.861336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.861365 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.861428 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.861955 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.861958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.861960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.861969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.861985 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.861992 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.862061 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.862103 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.862133 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:08:15.865125 5092 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:15.868119 5092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:15.879318 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.876442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44247 (local address 127.0.0.1:35896)
0924 20:08:15.877076 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.877123 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:15.877230 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:15.877780 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:15.877789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:15.877868 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:15.878445 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:15.878449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:15.878473 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:15.878543 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:15.878979 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:15.878988 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:15.878994 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:08:15.879114 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 20:08:15.879185 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":65,"threads_started":1}
I20260924 20:08:15.879411 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.876568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.0.115.131:44247)
0924 20:08:15.876918 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.876930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.877340 (+ 410us) server_negotiation.cc:407] Connection header received
0924 20:08:15.877402 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.877407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.877617 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.877771 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.878747 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.878755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.878766 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.878776 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.878817 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.878843 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.878936 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.879145 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.879227 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":150,"threads_started":1}
I20260924 20:08:15.880501 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.877134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39069 (local address 127.0.0.1:40062)
0924 20:08:15.877556 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.877569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:15.877636 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:15.878542 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:15.878545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:15.878559 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:15.879058 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:15.879060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:15.879071 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:15.879106 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:15.880257 (+ 1151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:15.880260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:15.880263 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:08:15.880352 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:08:15.880387 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":103,"threads_started":1}
I20260924 20:08:15.880838 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.877635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.0.115.129:39069)
0924 20:08:15.878089 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.878102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.878126 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:08:15.878186 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.878191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.878389 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.878824 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.880071 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.880077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.880085 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.880094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.880131 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.880158 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.880269 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.880399 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.880471 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":130,"threads_started":1}
I20260924 20:08:15.881125 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.876557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:45291 (local address 127.0.0.1:44632)
0924 20:08:15.877388 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:15.877404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:15.877483 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:15.879401 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:15.879405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:15.879422 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:15.879829 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:15.879832 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:15.879841 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:15.879866 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:15.880815 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:15.880817 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:15.880821 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:08:15.880887 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 20:08:15.880919 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":127,"threads_started":1}
I20260924 20:08:15.881506 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:15.877236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.0.115.130:45291)
0924 20:08:15.878353 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:15.878367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:15.878396 (+ 29us) server_negotiation.cc:407] Connection header received
0924 20:08:15.878454 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:15.878460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:15.878659 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:15.878796 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:15.880050 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:15.880057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:15.880068 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:15.880080 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:15.880132 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:15.880164 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:15.880269 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:15.881284 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0924 20:08:15.881354 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":111,"threads_started":1}
I20260924 20:08:15.882588 5477 tablet_service.cc:1467] Processing CreateTablet for tablet 58a2253d5dda43e6abc77d748e8bb3e4 (DEFAULT_TABLE table=test-table [id=7b4bfce448d64d418ff130860eb8baed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:15.883553 5477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a2253d5dda43e6abc77d748e8bb3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:15.883450 5215 tablet_service.cc:1467] Processing CreateTablet for tablet 58a2253d5dda43e6abc77d748e8bb3e4 (DEFAULT_TABLE table=test-table [id=7b4bfce448d64d418ff130860eb8baed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:15.884603 5215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a2253d5dda43e6abc77d748e8bb3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:15.889714 5560 tablet_bootstrap.cc:492] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Bootstrap starting.
I20260924 20:08:15.890550 5346 tablet_service.cc:1467] Processing CreateTablet for tablet 58a2253d5dda43e6abc77d748e8bb3e4 (DEFAULT_TABLE table=test-table [id=7b4bfce448d64d418ff130860eb8baed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:15.891520 5346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a2253d5dda43e6abc77d748e8bb3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:15.891914 5560 tablet_bootstrap.cc:654] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:15.892534 5561 tablet_bootstrap.cc:492] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7: Bootstrap starting.
I20260924 20:08:15.892881 5560 log.cc:824] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:15.894550 5561 tablet_bootstrap.cc:654] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:15.895092 5560 tablet_bootstrap.cc:492] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: No bootstrap required, opened a new log
I20260924 20:08:15.895272 5561 log.cc:824] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:15.895308 5560 ts_tablet_manager.cc:1397] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:08:15.896927 5561 tablet_bootstrap.cc:492] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7: No bootstrap required, opened a new log
I20260924 20:08:15.897128 5561 ts_tablet_manager.cc:1397] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:08:15.898710 5564 tablet_bootstrap.cc:492] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Bootstrap starting.
I20260924 20:08:15.900647 5560 raft_consensus.cc:348] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:15.900866 5560 raft_consensus.cc:374] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:15.900959 5560 raft_consensus.cc:729] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8236f6c9dbce4ea08bfb896b0946a5a5, State: Initialized, Role: FOLLOWER
I20260924 20:08:15.901243 5564 tablet_bootstrap.cc:654] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:15.901142 5561 raft_consensus.cc:348] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:15.901391 5561 raft_consensus.cc:374] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:15.901454 5561 raft_consensus.cc:729] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb67a11d4d584eb590124b5e5bfd06a7, State: Initialized, Role: FOLLOWER
I20260924 20:08:15.901465 5560 consensus_queue.cc:260] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:15.901925 5561 consensus_queue.cc:260] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:15.902287 5543 heartbeater.cc:503] Master 127.0.115.190:44775 was elected leader, sending a full tablet report...
I20260924 20:08:15.902591 5281 heartbeater.cc:503] Master 127.0.115.190:44775 was elected leader, sending a full tablet report...
I20260924 20:08:15.902611 5560 ts_tablet_manager.cc:1428] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:08:15.902863 5561 ts_tablet_manager.cc:1428] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:08:15.902077 5564 log.cc:824] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:15.905488 5564 tablet_bootstrap.cc:492] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: No bootstrap required, opened a new log
I20260924 20:08:15.906189 5564 ts_tablet_manager.cc:1397] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 20:08:15.909759 5564 raft_consensus.cc:348] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:15.910074 5564 raft_consensus.cc:374] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:15.910135 5564 raft_consensus.cc:729] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064372eeea2c45aeba02671708c583d6, State: Initialized, Role: FOLLOWER
I20260924 20:08:15.910552 5564 consensus_queue.cc:260] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:15.911144 5412 heartbeater.cc:503] Master 127.0.115.190:44775 was elected leader, sending a full tablet report...
I20260924 20:08:15.911459 5564 ts_tablet_manager.cc:1428] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 20:08:15.918821 5413 tablet.cc:2369] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:08:15.998171 5282 tablet.cc:2369] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:08:16.099002 5544 tablet.cc:2369] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:16.361181 5565 raft_consensus.cc:482] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:16.361488 5565 raft_consensus.cc:504] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:16.362911 5565 leader_election.cc:288] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 064372eeea2c45aeba02671708c583d6 (127.0.115.130:45291), fb67a11d4d584eb590124b5e5bfd06a7 (127.0.115.129:39069)
I20260924 20:08:16.365741 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.363898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:52027 (local address 127.0.115.129:39069)
0924 20:08:16.364111 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:16.364117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:16.364137 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:08:16.364269 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:16.364273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:16.364320 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:16.364433 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:16.365174 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:16.365178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:16.365180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:16.365260 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:16.365281 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:16.365297 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:16.365444 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:16.365564 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 20:08:16.365605 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 20:08:16.365764 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.363260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39069 (local address 127.0.115.131:52027)
0924 20:08:16.363525 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:16.363542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:16.363680 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:16.364462 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:16.364464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:16.364479 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:16.364995 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:16.364998 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:16.365013 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:16.365044 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:16.365429 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:16.365432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:16.365436 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:08:16.365561 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 20:08:16.365604 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":109,"threads_started":1}
I20260924 20:08:16.366909 5235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2253d5dda43e6abc77d748e8bb3e4" candidate_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" is_pre_election: true
I20260924 20:08:16.367189 5235 raft_consensus.cc:2454] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8236f6c9dbce4ea08bfb896b0946a5a5 in term 0.
I20260924 20:08:16.367626 5431 leader_election.cc:302] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8236f6c9dbce4ea08bfb896b0946a5a5, fb67a11d4d584eb590124b5e5bfd06a7; no voters:
I20260924 20:08:16.367877 5565 raft_consensus.cc:2781] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:16.367978 5565 raft_consensus.cc:482] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:16.368036 5565 raft_consensus.cc:3037] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:16.368520 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.363857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:56595 (local address 127.0.115.130:45291)
0924 20:08:16.364026 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:16.364035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:16.364780 (+ 745us) server_negotiation.cc:407] Connection header received
0924 20:08:16.364916 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:16.364921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:16.364966 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:16.365048 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:16.367699 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:16.367703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:16.367706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:16.367733 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:16.367766 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:16.367788 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:16.367911 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:16.368353 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 20:08:16.368400 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:08:16.368661 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.363312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:45291 (local address 127.0.115.131:56595)
0924 20:08:16.364661 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:16.364688 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:16.364792 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:16.365342 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:16.365359 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:16.365379 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:16.366866 (+ 1487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:16.366868 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:16.366879 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:16.366910 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:16.368229 (+ 1319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:16.368234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:16.368237 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:08:16.368358 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:08:16.368418 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"thread_start_us":76,"threads_started":1}
I20260924 20:08:16.369443 5565 raft_consensus.cc:504] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:16.369853 5565 leader_election.cc:288] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 064372eeea2c45aeba02671708c583d6 (127.0.115.130:45291), fb67a11d4d584eb590124b5e5bfd06a7 (127.0.115.129:39069)
I20260924 20:08:16.370257 5235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2253d5dda43e6abc77d748e8bb3e4" candidate_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb67a11d4d584eb590124b5e5bfd06a7"
I20260924 20:08:16.370406 5235 raft_consensus.cc:3037] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:16.370847 5366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2253d5dda43e6abc77d748e8bb3e4" candidate_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064372eeea2c45aeba02671708c583d6" is_pre_election: true
I20260924 20:08:16.370903 5365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2253d5dda43e6abc77d748e8bb3e4" candidate_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "064372eeea2c45aeba02671708c583d6"
I20260924 20:08:16.371135 5366 raft_consensus.cc:2454] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8236f6c9dbce4ea08bfb896b0946a5a5 in term 0.
I20260924 20:08:16.371811 5235 raft_consensus.cc:2454] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8236f6c9dbce4ea08bfb896b0946a5a5 in term 1.
I20260924 20:08:16.372130 5431 leader_election.cc:302] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8236f6c9dbce4ea08bfb896b0946a5a5, fb67a11d4d584eb590124b5e5bfd06a7; no voters: 064372eeea2c45aeba02671708c583d6
I20260924 20:08:16.372336 5565 raft_consensus.cc:2781] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:16.372588 5565 raft_consensus.cc:686] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 8236f6c9dbce4ea08bfb896b0946a5a5, State: Running, Role: LEADER
I20260924 20:08:16.372938 5565 consensus_queue.cc:237] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } }
I20260924 20:08:16.377365 5093 catalog_manager.cc:5495] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8236f6c9dbce4ea08bfb896b0946a5a5 (127.0.115.131). New cstate: current_term: 1 leader_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8236f6c9dbce4ea08bfb896b0946a5a5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:16.442409 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.440801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.0.115.131:44247)
0924 20:08:16.441038 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:16.441041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:16.441115 (+ 74us) server_negotiation.cc:407] Connection header received
0924 20:08:16.441245 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:16.441248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:16.441315 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:16.441398 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:16.442032 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:16.442035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:16.442037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:16.442060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:16.442078 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:16.442086 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:16.442186 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:16.442252 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 20:08:16.442285 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260924 20:08:16.448762 5365 raft_consensus.cc:3037] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:16.448897 5235 raft_consensus.cc:1261] T 58a2253d5dda43e6abc77d748e8bb3e4 P fb67a11d4d584eb590124b5e5bfd06a7 [term 1 FOLLOWER]: Refusing update from remote peer 8236f6c9dbce4ea08bfb896b0946a5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:16.449676 5565 consensus_queue.cc:1035] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb67a11d4d584eb590124b5e5bfd06a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:16.450421 5365 raft_consensus.cc:1261] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 1 FOLLOWER]: Refusing update from remote peer 8236f6c9dbce4ea08bfb896b0946a5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:16.451002 5565 consensus_queue.cc:1035] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "064372eeea2c45aeba02671708c583d6" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:16.458724 5582 mvcc.cc:205] Tried to move back new op lower bound from 7332988913446477824 to 7332988913147371520. Current Snapshot: MvccSnapshot[applied={T|T < 7332988913446477824}]
I20260924 20:08:16.468786 5091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:16.468986 5091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:16.471966 5091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 523d74c6619e4dc1bef93f833bd7d2a8: Sending DeleteTablet for 3 replicas of tablet 58a2253d5dda43e6abc77d748e8bb3e4
I20260924 20:08:16.472851 5346 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a2253d5dda43e6abc77d748e8bb3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:16 UTC) from {username='slave'} at 127.0.0.1:44632
I20260924 20:08:16.473003 5215 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a2253d5dda43e6abc77d748e8bb3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:16 UTC) from {username='slave'} at 127.0.0.1:40062
I20260924 20:08:16.473513 5591 tablet_replica.cc:330] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: stopping tablet replica
I20260924 20:08:16.473498 5477 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a2253d5dda43e6abc77d748e8bb3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:16 UTC) from {username='slave'} at 127.0.0.1:35896
I20260924 20:08:16.473912 5591 raft_consensus.cc:2229] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:16.474063 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5153
I20260924 20:08:16.474273 5591 raft_consensus.cc:2258] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:16.474152 5593 tablet_replica.cc:330] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: stopping tablet replica
I20260924 20:08:16.475991 5593 raft_consensus.cc:2229] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:16.476421 5593 raft_consensus.cc:2258] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:16.476663 5591 ts_tablet_manager.cc:1905] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:16.477772 5593 ts_tablet_manager.cc:1905] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:16.479532 5591 ts_tablet_manager.cc:1918] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:16.479643 5591 log.cc:1196] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/58a2253d5dda43e6abc77d748e8bb3e4
I20260924 20:08:16.479995 5591 ts_tablet_manager.cc:1939] T 58a2253d5dda43e6abc77d748e8bb3e4 P 064372eeea2c45aeba02671708c583d6: Deleting consensus metadata
I20260924 20:08:16.480726 5593 ts_tablet_manager.cc:1918] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:16.480823 5593 log.cc:1196] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/58a2253d5dda43e6abc77d748e8bb3e4
I20260924 20:08:16.481128 5593 ts_tablet_manager.cc:1939] T 58a2253d5dda43e6abc77d748e8bb3e4 P 8236f6c9dbce4ea08bfb896b0946a5a5: Deleting consensus metadata
I20260924 20:08:16.481412 5080 catalog_manager.cc:4841] TS 064372eeea2c45aeba02671708c583d6 (127.0.115.130:45291): tablet 58a2253d5dda43e6abc77d748e8bb3e4 (table test-table [id=7b4bfce448d64d418ff130860eb8baed]) successfully deleted
I20260924 20:08:16.482254 5078 catalog_manager.cc:4841] TS 8236f6c9dbce4ea08bfb896b0946a5a5 (127.0.115.131:44247): tablet 58a2253d5dda43e6abc77d748e8bb3e4 (table test-table [id=7b4bfce448d64d418ff130860eb8baed]) successfully deleted
W20260924 20:08:16.486299 5078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:39069 (error 108)
I20260924 20:08:16.487182 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5284
I20260924 20:08:16.488597 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.487089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39069 (local address 127.0.0.1:40074)
0924 20:08:16.488406 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:16.488488 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:39069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
W20260924 20:08:16.488744 5078 catalog_manager.cc:4584] TS fb67a11d4d584eb590124b5e5bfd06a7 (127.0.115.129:39069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58a2253d5dda43e6abc77d748e8bb3e4: Network error: Client connection negotiation failed: client connection to 127.0.115.129:39069: connect: Connection refused (error 111)
I20260924 20:08:16.493963 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5415
I20260924 20:08:16.499363 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5062
2026-09-24T20:08:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1447 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 20:08:16.521291 462 mini_oidc.cc:152] Starting JWKS server
I20260924 20:08:16.522876 462 webserver.cc:460] Webserver started at https://127.0.0.1:34821/ using document root <none> and password file <none>
I20260924 20:08:16.523017 462 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:08:16.523236 462 webserver.cc:460] Webserver started at http://127.0.0.1:39981/ using document root <none> and password file <none>
2026-09-24T20:08:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:16Z Disabled control of system clock
I20260924 20:08:16.532343 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46211
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:46211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34821/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/testchainca.cert with env {}
W20260924 20:08:16.646770 5601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:16.647042 5601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:16.647082 5601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:16.651135 5601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:16.651203 5601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:16.651218 5601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:16.651237 5601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:16.651252 5601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:08:16.651268 5601 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:08:16.651281 5601 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34821/jwks.json
I20260924 20:08:16.656703 5601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:46211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34821/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:16.657814 5601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:16.658820 5601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:16.667249 5606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:16.667358 5609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:16.667256 5607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:16.667511 5601 server_base.cc:1034] running on GCE node
I20260924 20:08:16.668157 5601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:16.669318 5601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:16.670542 5601 hybrid_clock.cc:648] HybridClock initialized: now 1790280496670547 us; error 61 us; skew 500 ppm
I20260924 20:08:16.672675 5601 webserver.cc:460] Webserver started at http://127.0.115.190:43377/ using document root <none> and password file <none>
I20260924 20:08:16.673287 5601 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:16.673383 5601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:16.673636 5601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:16.675771 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "2ba5b32cebbd40fea50787f1987a10d1"
format_stamp: "Formatted at 2026-09-24 20:08:16 on dist-test-slave-2pm7"
server_key: "7ef701235f86a93685932bbcdba21ea0"
server_key_iv: "6f5881973865185389bba55cf8eaa9e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:16.676436 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "2ba5b32cebbd40fea50787f1987a10d1"
format_stamp: "Formatted at 2026-09-24 20:08:16 on dist-test-slave-2pm7"
server_key: "7ef701235f86a93685932bbcdba21ea0"
server_key_iv: "6f5881973865185389bba55cf8eaa9e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:16.680316 5601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:08:16.682741 5615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:16.683996 5601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:16.684142 5601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "2ba5b32cebbd40fea50787f1987a10d1"
format_stamp: "Formatted at 2026-09-24 20:08:16 on dist-test-slave-2pm7"
server_key: "7ef701235f86a93685932bbcdba21ea0"
server_key_iv: "6f5881973865185389bba55cf8eaa9e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:16.684252 5601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:16.710551 5601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:16.711364 5601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:16.711586 5601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:16.719168 5667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:46211 every 8 connection(s)
I20260924 20:08:16.719172 5601 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:46211
I20260924 20:08:16.720311 5601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:16.724300 5668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:16.728436 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5601
I20260924 20:08:16.728582 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:16.728883 462 external_mini_cluster.cc:1428] Setting key 54dd2b0975ac831cafb90196f188348a
I20260924 20:08:16.731137 5668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1: Bootstrap starting.
I20260924 20:08:16.733790 5668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:16.734568 5668 log.cc:824] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:16.736848 5668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1: No bootstrap required, opened a new log
I20260924 20:08:16.740559 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.730444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.0.115.190:46211)
0924 20:08:16.731167 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:16.731184 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:16.731254 (+ 70us) server_negotiation.cc:407] Connection header received
0924 20:08:16.732066 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:16.732089 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:16.732537 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:16.732854 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:16.733318 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:16.734559 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:16.735674 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:16.736287 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:16.736415 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:16.737439 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:16.737445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:16.737459 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:16.738141 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:16.738256 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:16.738342 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:16.738496 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:16.738713 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 20:08:16.739276 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":171,"threads_started":1}
I20260924 20:08:16.741284 5668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5b32cebbd40fea50787f1987a10d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46211 } }
I20260924 20:08:16.741631 5668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:16.741688 5668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba5b32cebbd40fea50787f1987a10d1, State: Initialized, Role: FOLLOWER
I20260924 20:08:16.742260 5668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5b32cebbd40fea50787f1987a10d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46211 } }
I20260924 20:08:16.742424 5668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:16.742482 5668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:16.742573 5668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:16.743767 5668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5b32cebbd40fea50787f1987a10d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46211 } }
I20260924 20:08:16.744179 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:08:16.744160 5668 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba5b32cebbd40fea50787f1987a10d1; no voters:
I20260924 20:08:16.744522 5668 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:16.744602 5673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:16.744817 5673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba5b32cebbd40fea50787f1987a10d1, State: Running, Role: LEADER
I20260924 20:08:16.745296 5673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5b32cebbd40fea50787f1987a10d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46211 } }
I20260924 20:08:16.745884 5668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:16.746527 5673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba5b32cebbd40fea50787f1987a10d1. Latest consensus state: current_term: 1 leader_uuid: "2ba5b32cebbd40fea50787f1987a10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5b32cebbd40fea50787f1987a10d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46211 } } }
I20260924 20:08:16.746659 5673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:16.747052 5680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:16.747380 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba5b32cebbd40fea50787f1987a10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5b32cebbd40fea50787f1987a10d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46211 } } }
I20260924 20:08:16.748208 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:16.750540 5680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:16.757014 5680 catalog_manager.cc:1347] Generated new cluster ID: 5f3e482f2bec4716b563bdca3f0476e9
I20260924 20:08:16.757110 5680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:16.766949 5680 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:16.767899 5680 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:16.779959 5680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba5b32cebbd40fea50787f1987a10d1: Generated new TSK 0
I20260924 20:08:16.780849 5680 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 20:08:16.810683 5698 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 20:08:16.810817 5698 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 20:08:16.811455 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:16.793642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.0.115.190:46211)
0924 20:08:16.793792 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:16.793798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:16.793985 (+ 187us) server_negotiation.cc:407] Connection header received
0924 20:08:16.794195 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:16.794199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:16.794271 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:16.794390 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:08:16.794699 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:16.795554 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:16.796527 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:16.796900 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:16.797008 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:16.797378 (+ 370us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:08:16.811000 (+ 13622us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 20:08:16.811305 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 20:08:16.811540 5671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 20:08:16.812719 5697 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:16.793470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46211 (local address 127.0.0.1:50500)
0924 20:08:16.793902 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:16.793915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:16.794029 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:16.794407 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:16.794409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:16.794416 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:08:16.794574 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:16.794580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:16.795690 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:16.795693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:16.796399 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:16.796409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:16.797025 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:16.797028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:16.797093 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:16.797159 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:08:16.811258 (+ 14099us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 20:08:16.811478 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.115.190:46211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":149,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.115.190:46211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 20:08:16.813592 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5601
2026-09-24T20:08:16Z chronyd exiting
I20260924 20:08:16.852166 462 test_util.cc:175] -----------------------------------------------
I20260924 20:08:16.852279 462 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790280466534371-462-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 20:08:16.853348 462 mini_oidc.cc:152] Starting JWKS server
I20260924 20:08:16.854270 462 webserver.cc:460] Webserver started at https://127.0.0.1:39773/ using document root <none> and password file <none>
I20260924 20:08:16.854405 462 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:08:16.854681 462 webserver.cc:460] Webserver started at http://127.0.0.1:38149/ using document root <none> and password file <none>
2026-09-24T20:08:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:16Z Disabled control of system clock
I20260924 20:08:16.863822 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35905
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:35905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39773/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/testchainca.cert with env {}
W20260924 20:08:16.977217 5704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:16.977511 5704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:16.977689 5704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:16.981948 5704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:16.982009 5704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:16.982025 5704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:16.982043 5704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:16.982106 5704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:08:16.982128 5704 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:08:16.982157 5704 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39773/jwks.json
I20260924 20:08:16.987761 5704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:35905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39773/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:16.988955 5704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:16.990016 5704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:16.998778 5710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:16.998783 5712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:16.998762 5709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:16.998991 5704 server_base.cc:1034] running on GCE node
I20260924 20:08:16.999632 5704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:17.000937 5704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:17.002134 5704 hybrid_clock.cc:648] HybridClock initialized: now 1790280497002103 us; error 51 us; skew 500 ppm
I20260924 20:08:17.004031 5704 webserver.cc:460] Webserver started at http://127.0.115.190:39325/ using document root <none> and password file <none>
I20260924 20:08:17.004624 5704 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:17.004719 5704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:17.004952 5704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:17.007023 5704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "03bbd2a6c7574875acc9d1c5780b4141"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "a4bf1b5b2f60ccc8ac482553d061435c"
server_key_iv: "2c6d8ec652ea3c164c408df25b118082"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.007602 5704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "03bbd2a6c7574875acc9d1c5780b4141"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "a4bf1b5b2f60ccc8ac482553d061435c"
server_key_iv: "2c6d8ec652ea3c164c408df25b118082"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.011261 5704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:08:17.013799 5718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:17.015057 5704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:08:17.015208 5704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "03bbd2a6c7574875acc9d1c5780b4141"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "a4bf1b5b2f60ccc8ac482553d061435c"
server_key_iv: "2c6d8ec652ea3c164c408df25b118082"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.015324 5704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:17.051039 5704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:17.051826 5704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:17.052059 5704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:17.059623 5704 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:35905
I20260924 20:08:17.059728 5770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:35905 every 8 connection(s)
I20260924 20:08:17.060680 5704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:17.063987 5771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:17.069537 5771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141: Bootstrap starting.
I20260924 20:08:17.070052 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5704
I20260924 20:08:17.070209 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:17.070475 462 external_mini_cluster.cc:1428] Setting key 8e953171054ae6e286620f79fa4b6976
I20260924 20:08:17.072305 5771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:17.073056 5771 log.cc:824] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:17.075191 5771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141: No bootstrap required, opened a new log
I20260924 20:08:17.079589 5771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bbd2a6c7574875acc9d1c5780b4141" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35905 } }
I20260924 20:08:17.079825 5771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:17.079860 5771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bbd2a6c7574875acc9d1c5780b4141, State: Initialized, Role: FOLLOWER
I20260924 20:08:17.080338 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:17.071830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.0.115.190:35905)
0924 20:08:17.072564 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:17.072580 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:17.072614 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:08:17.073316 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:17.073335 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:17.073749 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:17.074029 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:17.074381 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.075403 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.076629 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.077117 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.077254 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.078137 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:17.078142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:17.078158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:17.078622 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:17.078686 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:17.078709 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:17.078795 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:17.078998 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 20:08:17.079416 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":187,"threads_started":1}
I20260924 20:08:17.080389 5771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bbd2a6c7574875acc9d1c5780b4141" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35905 } }
I20260924 20:08:17.080559 5771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:17.080628 5771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:17.080703 5771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:17.081676 5771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bbd2a6c7574875acc9d1c5780b4141" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35905 } }
I20260924 20:08:17.082029 5771 leader_election.cc:302] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bbd2a6c7574875acc9d1c5780b4141; no voters:
I20260924 20:08:17.082372 5771 leader_election.cc:288] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:17.082466 5776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:17.082708 5776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [term 1 LEADER]: Becoming Leader. State: Replica: 03bbd2a6c7574875acc9d1c5780b4141, State: Running, Role: LEADER
I20260924 20:08:17.083019 5776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bbd2a6c7574875acc9d1c5780b4141" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35905 } }
I20260924 20:08:17.083465 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:08:17.083725 5771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:17.084802 5777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03bbd2a6c7574875acc9d1c5780b4141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bbd2a6c7574875acc9d1c5780b4141" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35905 } } }
I20260924 20:08:17.084924 5777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:17.085268 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bbd2a6c7574875acc9d1c5780b4141. Latest consensus state: current_term: 1 leader_uuid: "03bbd2a6c7574875acc9d1c5780b4141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bbd2a6c7574875acc9d1c5780b4141" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35905 } } }
I20260924 20:08:17.085383 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:17.085686 5785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:17.088697 5785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:17.094671 5785 catalog_manager.cc:1347] Generated new cluster ID: 9fd832c3051b4d2d92b873bb8abb271d
I20260924 20:08:17.094763 5785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:17.116160 5785 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:17.117828 5785 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:17.123322 5785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03bbd2a6c7574875acc9d1c5780b4141: Generated new TSK 0
I20260924 20:08:17.124135 5785 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 20:08:17.146566 5801 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 20:08:17.146652 5801 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 20:08:17.147281 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:17.131553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.0.115.190:35905)
0924 20:08:17.131699 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:17.131704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:17.131897 (+ 193us) server_negotiation.cc:407] Connection header received
0924 20:08:17.132090 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:17.132092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:17.132139 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:17.132231 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:08:17.132491 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.133033 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.134286 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.134498 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.134570 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.134905 (+ 335us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:08:17.146952 (+ 12047us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 20:08:17.147164 (+ 212us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 20:08:17.147363 5774 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 20:08:17.147607 5800 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:17.131501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35905 (local address 127.0.0.1:45838)
0924 20:08:17.131803 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:17.131818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:17.131966 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:17.132258 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:17.132261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:17.132270 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:08:17.132386 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:17.132393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.133152 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.133156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:17.134168 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:17.134179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.134618 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.134622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:17.134714 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.134741 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:08:17.147175 (+ 12434us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 20:08:17.147373 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.115.190:35905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260924 20:08:17.148433 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5704
2026-09-24T20:08:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 20:08:17.188611 462 mini_oidc.cc:152] Starting JWKS server
I20260924 20:08:17.189445 462 webserver.cc:460] Webserver started at https://127.0.0.1:39503/ using document root <none> and password file <none>
I20260924 20:08:17.189527 462 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:08:17.189739 462 webserver.cc:460] Webserver started at http://127.0.0.1:45343/ using document root <none> and password file <none>
2026-09-24T20:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:17Z Disabled control of system clock
I20260924 20:08:17.198757 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34653
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:34653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39503/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 20:08:17.312474 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:17.312819 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:17.312911 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:17.317157 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:17.317257 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:17.317284 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:17.317322 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:17.317361 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:08:17.317397 5807 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:08:17.317430 5807 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39503/jwks.json
I20260924 20:08:17.322655 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:34653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39503/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:17.323844 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:17.325336 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:17.334440 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:17.334434 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:17.334434 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:17.334479 5807 server_base.cc:1034] running on GCE node
I20260924 20:08:17.335434 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:17.336676 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:17.337841 5807 hybrid_clock.cc:648] HybridClock initialized: now 1790280497337806 us; error 52 us; skew 500 ppm
I20260924 20:08:17.339859 5807 webserver.cc:460] Webserver started at http://127.0.115.190:38787/ using document root <none> and password file <none>
I20260924 20:08:17.340502 5807 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:17.340569 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:17.340741 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:17.342818 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "736f2344631e44c39f446d4da62c4ccf"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "70c7f969cabb98ffaaabcdf8ce934813"
server_key_iv: "e39535053de02ef1697e9042e51ef386"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.343359 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "736f2344631e44c39f446d4da62c4ccf"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "70c7f969cabb98ffaaabcdf8ce934813"
server_key_iv: "e39535053de02ef1697e9042e51ef386"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.347163 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:08:17.349759 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:17.351069 5807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:17.351228 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "736f2344631e44c39f446d4da62c4ccf"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "70c7f969cabb98ffaaabcdf8ce934813"
server_key_iv: "e39535053de02ef1697e9042e51ef386"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.351354 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:17.361304 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:17.361991 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:17.362197 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:17.369345 5807 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:34653
I20260924 20:08:17.369357 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:34653 every 8 connection(s)
I20260924 20:08:17.370361 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:17.374017 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:17.374421 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5807
I20260924 20:08:17.374549 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:17.374894 462 external_mini_cluster.cc:1428] Setting key 5aedd343e091b2d58081e7d2e4b96239
I20260924 20:08:17.380591 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf: Bootstrap starting.
I20260924 20:08:17.383291 5874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:17.384215 5874 log.cc:824] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:17.384636 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:17.376433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.0.115.190:34653)
0924 20:08:17.377075 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:17.377088 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:17.377126 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:08:17.377784 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:17.377802 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:17.378203 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:17.378490 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:17.378915 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.379903 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.381102 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.381589 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.381776 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.382713 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:17.382719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:17.382732 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:17.383116 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:17.383169 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:17.383193 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:17.383271 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:17.383481 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 20:08:17.383824 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":128,"threads_started":1}
I20260924 20:08:17.386488 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf: No bootstrap required, opened a new log
I20260924 20:08:17.387578 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:08:17.390547 5874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f2344631e44c39f446d4da62c4ccf" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34653 } }
I20260924 20:08:17.390802 5874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:17.390894 5874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736f2344631e44c39f446d4da62c4ccf, State: Initialized, Role: FOLLOWER
I20260924 20:08:17.391381 5874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f2344631e44c39f446d4da62c4ccf" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34653 } }
I20260924 20:08:17.391541 5874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:17.391630 5874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:17.391733 5874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:17.392750 5874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f2344631e44c39f446d4da62c4ccf" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34653 } }
I20260924 20:08:17.393111 5874 leader_election.cc:302] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736f2344631e44c39f446d4da62c4ccf; no voters:
I20260924 20:08:17.393457 5874 leader_election.cc:288] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:17.393675 5879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:17.393926 5879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [term 1 LEADER]: Becoming Leader. State: Replica: 736f2344631e44c39f446d4da62c4ccf, State: Running, Role: LEADER
I20260924 20:08:17.394235 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f2344631e44c39f446d4da62c4ccf" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34653 } }
I20260924 20:08:17.394847 5874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:17.396061 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736f2344631e44c39f446d4da62c4ccf. Latest consensus state: current_term: 1 leader_uuid: "736f2344631e44c39f446d4da62c4ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f2344631e44c39f446d4da62c4ccf" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34653 } } }
I20260924 20:08:17.396201 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:17.396469 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736f2344631e44c39f446d4da62c4ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736f2344631e44c39f446d4da62c4ccf" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34653 } } }
I20260924 20:08:17.396586 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:17.396703 5888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:17.400070 5888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:17.406239 5888 catalog_manager.cc:1347] Generated new cluster ID: cde5b8c5ebaa4be7be1fe7dc323587e5
I20260924 20:08:17.406330 5888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:17.428223 5888 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:17.429631 5888 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:17.438258 5888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736f2344631e44c39f446d4da62c4ccf: Generated new TSK 0
I20260924 20:08:17.439162 5888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:08:17.461923 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:17.455341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.0.115.190:34653)
0924 20:08:17.455496 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:17.455501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:17.455752 (+ 251us) server_negotiation.cc:407] Connection header received
0924 20:08:17.455915 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:17.455919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:17.456019 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:17.456133 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:08:17.456488 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.457341 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.458502 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.458860 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.458964 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.459315 (+ 351us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:08:17.461530 (+ 2215us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 20:08:17.461777 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 20:08:17.461999 5877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 20:08:17.463945 5903 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:17.455254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:34653 (local address 127.0.0.1:41166)
0924 20:08:17.455654 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:17.455668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:17.455780 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:17.456241 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:17.456244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:17.456255 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:08:17.456371 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:17.456376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.457488 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.457492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:17.458347 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:17.458357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.458998 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.459002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:17.459077 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.459105 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:08:17.461714 (+ 2609us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 20:08:17.463763 (+ 2049us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.115.190:34653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":1744,"thread_start_us":107,"threads_started":1}
I20260924 20:08:17.464529 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5807
E20260924 20:08:17.473492 5904 webserver.cc:533] Webserver: sslize failed:
E20260924 20:08:17.473582 5904 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T20:08:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T20:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:17Z Disabled control of system clock
I20260924 20:08:17.512508 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:32933
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:32933
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/testchainca.cert with env {}
W20260924 20:08:17.626842 5908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:17.627174 5908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:17.627245 5908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:17.631330 5908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:17.631392 5908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:17.631407 5908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:17.631423 5908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:17.631439 5908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:08:17.631486 5908 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:08:17.631515 5908 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 20:08:17.637077 5908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:32933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:32933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:17.638224 5908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:17.639298 5908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:17.648124 5913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:17.648124 5916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:17.648152 5914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:17.648497 5908 server_base.cc:1034] running on GCE node
I20260924 20:08:17.649070 5908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:17.650265 5908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:17.651428 5908 hybrid_clock.cc:648] HybridClock initialized: now 1790280497651408 us; error 31 us; skew 500 ppm
I20260924 20:08:17.653349 5908 webserver.cc:460] Webserver started at http://127.0.115.190:33721/ using document root <none> and password file <none>
I20260924 20:08:17.653949 5908 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:17.654013 5908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:17.654232 5908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:17.656351 5908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "37faed2208b440ab8d6b97900811a7d5"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "709020adc361e7e69883660b0568df6e"
server_key_iv: "869d72ebfd1555f16d15990b6a209a12"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.656929 5908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "37faed2208b440ab8d6b97900811a7d5"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "709020adc361e7e69883660b0568df6e"
server_key_iv: "869d72ebfd1555f16d15990b6a209a12"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.660677 5908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:08:17.663146 5922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:17.664434 5908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 20:08:17.664583 5908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "37faed2208b440ab8d6b97900811a7d5"
format_stamp: "Formatted at 2026-09-24 20:08:17 on dist-test-slave-2pm7"
server_key: "709020adc361e7e69883660b0568df6e"
server_key_iv: "869d72ebfd1555f16d15990b6a209a12"
server_key_version: "encryptionkey@0"
I20260924 20:08:17.664703 5908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:17.717649 5908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:17.718457 5908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:17.718673 5908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:17.726797 5974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:32933 every 8 connection(s)
I20260924 20:08:17.726801 5908 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:32933
I20260924 20:08:17.727890 5908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:17.728935 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 5908
I20260924 20:08:17.729061 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:17.729393 462 external_mini_cluster.cc:1428] Setting key 5aba0a87e94bcdccb2a94c212f42f544
I20260924 20:08:17.731830 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:17.737712 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5: Bootstrap starting.
I20260924 20:08:17.739146 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:17.730892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.0.115.190:32933)
0924 20:08:17.731571 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:17.731586 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:17.731623 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:08:17.732399 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:17.732419 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:17.732837 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:17.733181 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:17.733635 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.734614 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.735502 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.736067 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.736234 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.737216 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:17.737220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:17.737234 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:17.737611 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:17.737666 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:17.737696 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:17.737807 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:17.737957 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 20:08:17.738233 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":153,"threads_started":1}
I20260924 20:08:17.740475 5975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:17.741343 5975 log.cc:824] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:17.742234 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:08:17.743813 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5: No bootstrap required, opened a new log
I20260924 20:08:17.748296 5975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37faed2208b440ab8d6b97900811a7d5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 32933 } }
I20260924 20:08:17.748519 5975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:17.748564 5975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37faed2208b440ab8d6b97900811a7d5, State: Initialized, Role: FOLLOWER
I20260924 20:08:17.748993 5975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37faed2208b440ab8d6b97900811a7d5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 32933 } }
I20260924 20:08:17.749114 5975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:17.749182 5975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:17.749286 5975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:17.750176 5975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37faed2208b440ab8d6b97900811a7d5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 32933 } }
I20260924 20:08:17.750550 5975 leader_election.cc:302] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37faed2208b440ab8d6b97900811a7d5; no voters:
I20260924 20:08:17.750910 5975 leader_election.cc:288] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:17.751037 5980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:17.751300 5980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [term 1 LEADER]: Becoming Leader. State: Replica: 37faed2208b440ab8d6b97900811a7d5, State: Running, Role: LEADER
I20260924 20:08:17.751611 5980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37faed2208b440ab8d6b97900811a7d5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 32933 } }
I20260924 20:08:17.752198 5975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:17.753393 5982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37faed2208b440ab8d6b97900811a7d5. Latest consensus state: current_term: 1 leader_uuid: "37faed2208b440ab8d6b97900811a7d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37faed2208b440ab8d6b97900811a7d5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 32933 } } }
I20260924 20:08:17.753551 5982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:17.753906 5981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37faed2208b440ab8d6b97900811a7d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37faed2208b440ab8d6b97900811a7d5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 32933 } } }
I20260924 20:08:17.753975 5989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:17.754588 5981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:17.757426 5989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:17.763216 5989 catalog_manager.cc:1347] Generated new cluster ID: 4a9dc9798ce64455a6ae4b8ac5a5ac6d
I20260924 20:08:17.763290 5989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:17.772531 5989 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:17.773759 5989 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:17.779656 5989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37faed2208b440ab8d6b97900811a7d5: Generated new TSK 0
I20260924 20:08:17.780491 5989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:08:17.860034 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:17.792068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.0.115.190:32933)
0924 20:08:17.792210 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:17.792215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:17.792386 (+ 171us) server_negotiation.cc:407] Connection header received
0924 20:08:17.792519 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:17.792523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:17.792579 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:17.792708 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:08:17.792937 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.793783 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.794660 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.795022 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.795129 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.795451 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:08:17.859479 (+ 64028us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 20:08:17.859830 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41988: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 20:08:17.860129 5978 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41988: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 20:08:17.860220 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:17.791973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:32933 (local address 127.0.0.1:41988)
0924 20:08:17.792299 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:17.792311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:17.792406 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:17.792720 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:17.792723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:17.792730 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:08:17.792828 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:17.792833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.793906 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.793908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:17.794542 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:17.794549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:17.795149 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:17.795152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:17.795240 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:17.795263 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:08:17.859750 (+ 64487us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 20:08:17.859988 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.115.190:32933: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260924 20:08:17.860930 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 5908
2026-09-24T20:08:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (376 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:18 dist-test-slave-2pm7 krb5kdc[6016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:18 dist-test-slave-2pm7 krb5kdc[6016](info): set up 2 sockets
Sep 24 20:08:18 dist-test-slave-2pm7 krb5kdc[6016](info): commencing operation
krb5kdc: starting...
W20260924 20:08:19.172555 462 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:19 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:19.298694 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35749
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:19.410446 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:19.410763 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:19.410853 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:19.414968 6029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:19.415092 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:19.415126 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:19.415158 6029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:19.415194 6029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:19.420661 6029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:19.421833 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:19.422927 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:19.432122 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:19.432134 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:19.432134 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:19.432194 6029 server_base.cc:1034] running on GCE node
I20260924 20:08:19.432868 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:19.434103 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:19.435276 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790280499435248 us; error 41 us; skew 500 ppm
Sep 24 20:08:19 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:19.448298 6029 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:19.449280 6029 webserver.cc:460] Webserver started at http://127.0.115.190:46477/ using document root <none> and password file <none>
I20260924 20:08:19.449841 6029 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:19.449903 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:19.450076 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:19.452327 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "6c152b7b4a6549cf8428e40b44e45da0"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "e4b065b0ba0543e9de8f7e10c34c9088"
server_key_iv: "f10d33c11c576a729a4a89b1ab6a2734"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.452850 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "6c152b7b4a6549cf8428e40b44e45da0"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "e4b065b0ba0543e9de8f7e10c34c9088"
server_key_iv: "f10d33c11c576a729a4a89b1ab6a2734"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.456755 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260924 20:08:19.459395 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:19.460788 6029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:08:19.460933 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "6c152b7b4a6549cf8428e40b44e45da0"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "e4b065b0ba0543e9de8f7e10c34c9088"
server_key_iv: "f10d33c11c576a729a4a89b1ab6a2734"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.461053 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:19.485671 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:19.490073 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:19.490322 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:19.498329 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:35749
I20260924 20:08:19.498661 6096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:35749 every 8 connection(s)
I20260924 20:08:19.499351 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:19.502287 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:19.505203 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6029
I20260924 20:08:19.505379 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:19.505790 462 external_mini_cluster.cc:1428] Setting key ce9a4f9a902f69c3f4a5543ae966baa2
I20260924 20:08:19.508432 6097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0: Bootstrap starting.
I20260924 20:08:19.511026 6097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:19.511961 6097 log.cc:824] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:19.514339 6097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0: No bootstrap required, opened a new log
Sep 24 20:08:19 dist-test-slave-2pm7 krb5kdc[6016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:19.518505 6097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152b7b4a6549cf8428e40b44e45da0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35749 } }
I20260924 20:08:19.518807 6097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:19.518863 6097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c152b7b4a6549cf8428e40b44e45da0, State: Initialized, Role: FOLLOWER
I20260924 20:08:19.519419 6097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152b7b4a6549cf8428e40b44e45da0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35749 } }
I20260924 20:08:19.519587 6097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:19.519675 6097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:19.519788 6097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:19.520771 6097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152b7b4a6549cf8428e40b44e45da0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35749 } }
I20260924 20:08:19.521152 6097 leader_election.cc:302] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c152b7b4a6549cf8428e40b44e45da0; no voters:
I20260924 20:08:19.521497 6097 leader_election.cc:288] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:19.521595 6102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:19.521823 6102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [term 1 LEADER]: Becoming Leader. State: Replica: 6c152b7b4a6549cf8428e40b44e45da0, State: Running, Role: LEADER
I20260924 20:08:19.522210 6102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152b7b4a6549cf8428e40b44e45da0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35749 } }
I20260924 20:08:19.522689 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:19.507213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.0.115.190:35749)
0924 20:08:19.507658 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:19.507676 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:19.507714 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:08:19.508476 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:19.508497 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:19.508900 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:19.509210 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:19.510171 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:19.511355 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:19.512545 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:19.513209 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:19.513344 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:19.516224 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:19.516232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:19.516251 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:19.516285 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:19.518938 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:19.519448 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:19.519453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:19.519459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:19.519616 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:19.520017 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:19.520020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:19.520022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:19.520473 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:19.520643 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:19.521026 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 20:08:19.521402 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":141,"threads_started":1}
I20260924 20:08:19.522799 6097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:19.523530 6104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c152b7b4a6549cf8428e40b44e45da0. Latest consensus state: current_term: 1 leader_uuid: "6c152b7b4a6549cf8428e40b44e45da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152b7b4a6549cf8428e40b44e45da0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35749 } } }
I20260924 20:08:19.523829 6104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:19.524120 6103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c152b7b4a6549cf8428e40b44e45da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152b7b4a6549cf8428e40b44e45da0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35749 } } }
I20260924 20:08:19.524235 6103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0 [sys.catalog]: This master's current role is: LEADER
W20260924 20:08:19.528420 6117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:08:19.528506 6117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:08:19.528586 6111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:19.529448 6111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:19.535316 6111 catalog_manager.cc:1347] Generated new cluster ID: b78fd37950804982b4206566db3a2a86
I20260924 20:08:19.535386 6111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:19.557106 6111 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:19.558408 6111 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:19.565802 6111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c152b7b4a6549cf8428e40b44e45da0: Generated new TSK 0
I20260924 20:08:19.566636 6111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:08:19.603960 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35749
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:19.720077 6123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:19.720374 6123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:19.720458 6123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:19.724774 6123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:19.724855 6123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:19.724947 6123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:19.730412 6123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:19.731560 6123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:19.732709 6123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:19.742653 6128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:19.742709 6129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:19.742796 6123 server_base.cc:1034] running on GCE node
W20260924 20:08:19.742966 6131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:19.743443 6123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:19.744190 6123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:19.745352 6123 hybrid_clock.cc:648] HybridClock initialized: now 1790280499745317 us; error 46 us; skew 500 ppm
Sep 24 20:08:19 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:19.748234 6123 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:08:19.749099 6123 webserver.cc:460] Webserver started at http://127.0.115.129:35281/ using document root <none> and password file <none>
I20260924 20:08:19.749703 6123 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:19.749770 6123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:19.749982 6123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:19.752418 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "ca17f8e64d364e8f96fcb7111a66b2a7"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "eccfda792e5e76119c5416a60ba39ee7"
server_key_iv: "ced6d9bb6060dbf50f1a73e73ecaea51"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.753216 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "ca17f8e64d364e8f96fcb7111a66b2a7"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "eccfda792e5e76119c5416a60ba39ee7"
server_key_iv: "ced6d9bb6060dbf50f1a73e73ecaea51"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.757371 6123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:08:19.759881 6138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:19.761348 6123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:19.761477 6123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "ca17f8e64d364e8f96fcb7111a66b2a7"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "eccfda792e5e76119c5416a60ba39ee7"
server_key_iv: "ced6d9bb6060dbf50f1a73e73ecaea51"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.761596 6123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:19.790134 6123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:19.793351 6123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:19.793567 6123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:19.794195 6123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:19.795047 6123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:19.795094 6123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:19.795167 6123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:19.795198 6123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:19.804644 6123 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:40259
I20260924 20:08:19.804674 6251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:40259 every 8 connection(s)
I20260924 20:08:19.805567 6123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:19.811347 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6123
I20260924 20:08:19.811477 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:19.811812 462 external_mini_cluster.cc:1428] Setting key c6e5f05304745c3bb67e3c8c2189b4cd
Sep 24 20:08:19 dist-test-slave-2pm7 krb5kdc[6016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:19.822043 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:19.807522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:54353 (local address 127.0.115.190:35749)
0924 20:08:19.807756 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:19.807761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:19.808753 (+ 992us) server_negotiation.cc:407] Connection header received
0924 20:08:19.810145 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:19.810153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:19.810209 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:19.810290 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:19.811830 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:19.812456 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:19.813996 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:19.814240 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:19.814326 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:19.817542 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:19.817545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:19.817547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:19.817570 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:19.819451 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:19.820157 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:19.820164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:19.820169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:19.820267 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:19.820736 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:19.820739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:19.820741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:19.820997 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:19.821109 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:19.821670 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 20:08:19.821827 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:08:19.822948 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:19.808058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35749 (local address 127.0.115.129:54353)
0924 20:08:19.808561 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:19.808616 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:19.809906 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:19.810481 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:19.810488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:19.810907 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:19.811645 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:19.811669 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:19.812583 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:19.812587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:19.813853 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:19.813867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:19.814406 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:19.814412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:19.814672 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:19.815472 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:19.815490 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:19.817355 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:19.819656 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:19.819661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:19.819675 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:19.820040 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:19.820397 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:19.820400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:19.820402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:19.820557 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:19.821125 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:19.821132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:19.821399 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 20:08:19.821567 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 20:08:19.821913 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":122,"threads_started":1}
I20260924 20:08:19.824604 6252 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35749
I20260924 20:08:19.825001 6252 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:08:19.825625 6252 heartbeater.cc:511] Master 127.0.115.190:35749 requested a full tablet report, sending...
I20260924 20:08:19.827597 6056 ts_manager.cc:194] Registered new tserver with Master: ca17f8e64d364e8f96fcb7111a66b2a7 (127.0.115.129:40259)
I20260924 20:08:19.829010 6056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:54353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:08:19.844521 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35749
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:19.956501 6257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:19.956811 6257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:19.956856 6257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:19.960943 6257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:19.961002 6257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:19.961100 6257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:19.966636 6257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:19.967787 6257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:19.968871 6257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:19.978927 6263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:19.979084 6262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:19.978926 6265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:19.979008 6257 server_base.cc:1034] running on GCE node
I20260924 20:08:19.979591 6257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:19.980347 6257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:19.981544 6257 hybrid_clock.cc:648] HybridClock initialized: now 1790280499981507 us; error 46 us; skew 500 ppm
Sep 24 20:08:19 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:19.984750 6257 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:08:19.985714 6257 webserver.cc:460] Webserver started at http://127.0.115.130:32935/ using document root <none> and password file <none>
I20260924 20:08:19.986289 6257 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:19.986348 6257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:19.986588 6257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:19.988710 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "88cf16349c014f038744d890b5699208"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "c509cbebadc04047a3f2794ff0b16a25"
server_key_iv: "af8b455d44d3975c969627c5c7b74c84"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.989267 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "88cf16349c014f038744d890b5699208"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "c509cbebadc04047a3f2794ff0b16a25"
server_key_iv: "af8b455d44d3975c969627c5c7b74c84"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.992996 6257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:08:19.995595 6272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:19.996924 6257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:19.997056 6257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "88cf16349c014f038744d890b5699208"
format_stamp: "Formatted at 2026-09-24 20:08:19 on dist-test-slave-2pm7"
server_key: "c509cbebadc04047a3f2794ff0b16a25"
server_key_iv: "af8b455d44d3975c969627c5c7b74c84"
server_key_version: "encryptionkey@0"
I20260924 20:08:19.997179 6257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:20.007824 6257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:20.010658 6257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:20.010866 6257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:20.011466 6257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:20.012425 6257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:20.012477 6257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:20.012549 6257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:20.012589 6257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:20.021315 6257 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:41289
I20260924 20:08:20.021346 6385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:41289 every 8 connection(s)
I20260924 20:08:20.022205 6257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:20.031380 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6257
I20260924 20:08:20.031504 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:20.031852 462 external_mini_cluster.cc:1428] Setting key ef23e1c187ea6a6d89d85365da9b400f
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:20.037487 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:20.024148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:40983 (local address 127.0.115.190:35749)
0924 20:08:20.024339 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.024346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.025409 (+ 1063us) server_negotiation.cc:407] Connection header received
0924 20:08:20.026844 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.026847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.026903 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.026983 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.028221 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.028892 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.030036 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.030278 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.030388 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:20.033303 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:20.033307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:20.033310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:20.033337 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:20.035177 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.035753 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.035759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.035762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.035860 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.036253 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.036258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.036262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.036512 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:20.036634 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:20.037100 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 20:08:20.037244 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:08:20.038369 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:20.024673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35749 (local address 127.0.115.130:40983)
0924 20:08:20.025270 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:20.025313 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:20.026631 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:20.027165 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:20.027173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:20.027521 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:20.028072 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:20.028088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.029016 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.029020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:20.029905 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:20.029914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.030439 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.030443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:20.030628 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:20.031188 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:20.031215 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:20.033115 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:20.035321 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:20.035331 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:20.035349 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:20.035637 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:20.035990 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:20.035995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:20.036001 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:20.036146 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:20.036614 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:20.036619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:20.036847 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 20:08:20.036994 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 20:08:20.037256 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
I20260924 20:08:20.039577 6386 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35749
I20260924 20:08:20.039871 6386 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:20.040400 6386 heartbeater.cc:511] Master 127.0.115.190:35749 requested a full tablet report, sending...
I20260924 20:08:20.041455 6056 ts_manager.cc:194] Registered new tserver with Master: 88cf16349c014f038744d890b5699208 (127.0.115.130:41289)
I20260924 20:08:20.042040 6056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:40983
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:08:20.061410 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35749
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:20.174263 6391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:20.174582 6391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:20.174643 6391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:20.178983 6391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:20.179039 6391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:20.179101 6391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:20.184255 6391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:20.185204 6391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:20.186209 6391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:20.195670 6399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:20.195719 6397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:20.195676 6396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:20.195850 6391 server_base.cc:1034] running on GCE node
I20260924 20:08:20.196355 6391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:20.196913 6391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:20.198091 6391 hybrid_clock.cc:648] HybridClock initialized: now 1790280500198052 us; error 55 us; skew 500 ppm
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.201067 6391 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:20.202258 6391 webserver.cc:460] Webserver started at http://127.0.115.131:46407/ using document root <none> and password file <none>
I20260924 20:08:20.202843 6391 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:20.202907 6391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:20.203071 6391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:20.205152 6391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "bda01bd0dbdd4a2992d3a16ac6393fe4"
format_stamp: "Formatted at 2026-09-24 20:08:20 on dist-test-slave-2pm7"
server_key: "70538c2d32bcf9634d5a45ef7ed8c515"
server_key_iv: "7cb2b391ed3b49c244da058f8068648f"
server_key_version: "encryptionkey@0"
I20260924 20:08:20.205709 6391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "bda01bd0dbdd4a2992d3a16ac6393fe4"
format_stamp: "Formatted at 2026-09-24 20:08:20 on dist-test-slave-2pm7"
server_key: "70538c2d32bcf9634d5a45ef7ed8c515"
server_key_iv: "7cb2b391ed3b49c244da058f8068648f"
server_key_version: "encryptionkey@0"
I20260924 20:08:20.209542 6391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:08:20.212102 6406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:20.213495 6391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:08:20.213624 6391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "bda01bd0dbdd4a2992d3a16ac6393fe4"
format_stamp: "Formatted at 2026-09-24 20:08:20 on dist-test-slave-2pm7"
server_key: "70538c2d32bcf9634d5a45ef7ed8c515"
server_key_iv: "7cb2b391ed3b49c244da058f8068648f"
server_key_version: "encryptionkey@0"
I20260924 20:08:20.213783 6391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:20.235184 6391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:20.238400 6391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:20.238620 6391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:20.239327 6391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:20.240299 6391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:20.240352 6391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:20.240428 6391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:20.240473 6391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:20.250116 6391 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44315
I20260924 20:08:20.250138 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44315 every 8 connection(s)
I20260924 20:08:20.251564 6391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:20.258442 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6391
I20260924 20:08:20.258584 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:20.258864 462 external_mini_cluster.cc:1428] Setting key 5a79a6071896d34967706fc554f2ef3f
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:20.265955 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:20.253503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:46935 (local address 127.0.115.190:35749)
0924 20:08:20.253704 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.253708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.254516 (+ 808us) server_negotiation.cc:407] Connection header received
0924 20:08:20.256014 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.256017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.256083 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.256170 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.257361 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.257911 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.259252 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.259559 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.259664 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:20.262138 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:20.262142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:20.262145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:20.262172 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:20.263642 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.264360 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.264363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.264365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.264447 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.264776 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.264778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.264779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.264965 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:20.265129 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:20.265710 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 20:08:20.265835 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 20:08:20.266835 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:20.253876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35749 (local address 127.0.115.131:46935)
0924 20:08:20.254352 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:20.254396 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:20.255757 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:20.256341 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:20.256348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:20.256715 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:20.257205 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:20.257219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.258050 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.258053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:20.259126 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:20.259137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.259702 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.259707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:20.259885 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:20.260551 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:20.260567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:20.261976 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:20.263780 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:20.263786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:20.263801 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:20.264190 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:20.264539 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:20.264542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:20.264543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:20.264697 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:20.265150 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:20.265155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:20.265381 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 20:08:20.265585 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 20:08:20.265898 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260924 20:08:20.268134 6520 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35749
I20260924 20:08:20.268390 6520 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:20.268883 6520 heartbeater.cc:511] Master 127.0.115.190:35749 requested a full tablet report, sending...
I20260924 20:08:20.269910 6056 ts_manager.cc:194] Registered new tserver with Master: bda01bd0dbdd4a2992d3a16ac6393fe4 (127.0.115.131:44315)
I20260924 20:08:20.270484 6056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:46935
I20260924 20:08:20.272825 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/test-admin.keytab.
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.288655 462 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 20:08:20.291471 6527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:20.290040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.0.115.129:40259)
0924 20:08:20.290374 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.290388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.290412 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:08:20.290462 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.290468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.290663 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.290783 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.291287 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50114: BlockingRecv error: recv got EOF from 127.0.0.1:50114 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":131,"threads_started":1}
W20260924 20:08:20.291929 462 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc)
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.293001 462 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.293955 462 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:08:20.304641 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:20.294830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.0.115.129:40259)
0924 20:08:20.294996 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.295001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.295142 (+ 141us) server_negotiation.cc:407] Connection header received
0924 20:08:20.295311 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.295315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.295369 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.295466 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.296096 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.297021 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.297948 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.298367 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.298493 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:20.301024 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:20.301033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:20.301046 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:20.301069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:20.302860 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.303287 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.303292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.303293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.303367 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.303599 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.303640 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.303641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.303877 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:20.304104 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:20.304270 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 20:08:20.304445 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 20:08:20.308760 6527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:20.307661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.0.115.129:40259)
0924 20:08:20.307799 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.307803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.307900 (+ 97us) server_negotiation.cc:407] Connection header received
0924 20:08:20.308093 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.308096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.308159 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.308244 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.308658 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51306: BlockingRecv error: recv got EOF from 127.0.0.1:51306 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 20:08:20.308995 462 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.310014 462 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.310935 462 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:08:20.319666 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:20.311808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.0.115.129:40259)
0924 20:08:20.311986 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.311991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.312005 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:20.312131 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.312133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.312169 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.312235 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.312752 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.313284 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.314086 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.314324 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.314409 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:20.316734 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:20.316737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:20.316739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:20.316761 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:20.318132 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.318512 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.318515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.318516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.318590 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.318829 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.318831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.318832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.318988 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:20.319147 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:20.319357 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 20:08:20.319479 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 20:08:20.322485 6527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:20.321420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.115.129:40259)
0924 20:08:20.321554 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.321563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.321594 (+ 31us) server_negotiation.cc:407] Connection header received
0924 20:08:20.321717 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.321720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.321754 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.321816 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.322359 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51328: BlockingRecv error: recv got EOF from 127.0.0.1:51328 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 20:08:20.322839 462 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790280466534371-462-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.324015 462 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:20.325029 462 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:08:20.333700 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:20.326006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.0.115.129:40259)
0924 20:08:20.326150 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:20.326153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:20.326168 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:20.326284 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:20.326286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:20.326321 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:20.326382 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:20.326828 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.327322 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.328209 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:20.328414 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:20.328491 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:20.330680 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:20.330683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:20.330685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:20.330706 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:20.332196 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.332588 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.332590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.332592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.332661 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:20.332922 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:20.332924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:20.332925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:20.333074 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:20.333182 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:20.333381 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 20:08:20.333501 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:08:20.334846 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6123
I20260924 20:08:20.341152 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6257
I20260924 20:08:20.346732 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6391
I20260924 20:08:20.352288 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6029
2026-09-24T20:08:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2241 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6540](info): set up 2 sockets
Sep 24 20:08:20 dist-test-slave-2pm7 krb5kdc[6540](info): commencing operation
krb5kdc: starting...
W20260924 20:08:21.421144 462 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:21 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab.
Entry for principal oryx/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab.
I20260924 20:08:21.549084 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44475
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab
--principal=oryx/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:21.661422 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:21.661777 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:21.661852 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:21.665985 6553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:21.666074 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:21.666112 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:21.666173 6553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:21.666210 6553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:21.671615 6553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab
--principal=oryx/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:21.672784 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:21.673863 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:21.682406 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:21.682406 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:21.682420 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:21.682650 6553 server_base.cc:1034] running on GCE node
I20260924 20:08:21.683346 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:21.684505 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:21.685705 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790280501685687 us; error 28 us; skew 500 ppm
Sep 24 20:08:21 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:21.688450 6553 init.cc:377] Logged in from keytab as oryx/127.0.115.190@KRBTEST.COM (short username oryx)
I20260924 20:08:21.689359 6553 webserver.cc:460] Webserver started at http://127.0.115.190:44641/ using document root <none> and password file <none>
I20260924 20:08:21.689990 6553 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:21.690080 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:21.690320 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:21.692452 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "ab3766a2ab094b89b7abed98b7412ac0"
format_stamp: "Formatted at 2026-09-24 20:08:21 on dist-test-slave-2pm7"
server_key: "d5c2d6968e748c0c3a1956071699c417"
server_key_iv: "ff4bc099afbda66fc25404a50cdb9a22"
server_key_version: "encryptionkey@0"
I20260924 20:08:21.693032 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "ab3766a2ab094b89b7abed98b7412ac0"
format_stamp: "Formatted at 2026-09-24 20:08:21 on dist-test-slave-2pm7"
server_key: "d5c2d6968e748c0c3a1956071699c417"
server_key_iv: "ff4bc099afbda66fc25404a50cdb9a22"
server_key_version: "encryptionkey@0"
I20260924 20:08:21.696810 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:08:21.699303 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:21.700659 6553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:21.700799 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "ab3766a2ab094b89b7abed98b7412ac0"
format_stamp: "Formatted at 2026-09-24 20:08:21 on dist-test-slave-2pm7"
server_key: "d5c2d6968e748c0c3a1956071699c417"
server_key_iv: "ff4bc099afbda66fc25404a50cdb9a22"
server_key_version: "encryptionkey@0"
I20260924 20:08:21.700914 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:21.725224 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:21.728272 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:21.728490 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:21.736124 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44475
I20260924 20:08:21.736449 6620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44475 every 8 connection(s)
I20260924 20:08:21.737152 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:21.740038 6621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:21.745548 6621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: Bootstrap starting.
I20260924 20:08:21.745962 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6553
I20260924 20:08:21.746122 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:21.746399 462 external_mini_cluster.cc:1428] Setting key ffe8fcbca45ea62610337c2d3cb3ee3d
I20260924 20:08:21.748276 6621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:21.749017 6621 log.cc:824] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:21.750976 6621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: No bootstrap required, opened a new log
I20260924 20:08:21.754607 6621 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } }
I20260924 20:08:21.754900 6621 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:21.754974 6621 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3766a2ab094b89b7abed98b7412ac0, State: Initialized, Role: FOLLOWER
Sep 24 20:08:21 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:21.755482 6621 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } }
I20260924 20:08:21.755663 6621 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:21.755734 6621 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:21.755846 6621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:21.756983 6621 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } }
I20260924 20:08:21.757339 6621 leader_election.cc:302] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab3766a2ab094b89b7abed98b7412ac0; no voters:
I20260924 20:08:21.757696 6621 leader_election.cc:288] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:21.757763 6626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:21.757962 6626 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 1 LEADER]: Becoming Leader. State: Replica: ab3766a2ab094b89b7abed98b7412ac0, State: Running, Role: LEADER
I20260924 20:08:21.758260 6626 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } }
I20260924 20:08:21.759573 6626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3766a2ab094b89b7abed98b7412ac0. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } }
I20260924 20:08:21.759702 6626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:21.759907 6626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } }
I20260924 20:08:21.760017 6626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:21.760623 6621 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:21.761821 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:21.747762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.0.115.190:44475)
0924 20:08:21.748187 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:21.748203 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:21.748238 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:08:21.748895 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:21.748912 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:21.749299 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:21.749635 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:21.750461 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:21.751502 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:21.752425 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:21.752908 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:21.753106 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:21.755889 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:21.755895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:21.755911 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:21.755963 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:21.757871 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:21.758339 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:21.758347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:21.758354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:21.758499 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:21.758783 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:21.758787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:21.758789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:21.759373 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:21.760328 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:21.760487 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 20:08:21.760819 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":141,"threads_started":1}
I20260924 20:08:21.763077 6630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:21.765865 6630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:21.771720 6630 catalog_manager.cc:1347] Generated new cluster ID: 440abd657feb47f5aa8c2bb870b600ef
I20260924 20:08:21.771821 6630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:21.793684 6630 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:21.795085 6630 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:21.805579 6630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: Generated new TSK 0
I20260924 20:08:21.806519 6630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab.
Entry for principal oryx/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab.
I20260924 20:08:21.842653 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44475
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab
--principal=oryx/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:21.954461 6647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:21.954763 6647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:21.954845 6647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:21.958801 6647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:21.958889 6647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:21.958981 6647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:21.964546 6647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab
--principal=oryx/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:21.965595 6647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:21.966665 6647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:21.976282 6653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:21.976259 6655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:21.976253 6652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:21.976688 6647 server_base.cc:1034] running on GCE node
I20260924 20:08:21.977114 6647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:21.977675 6647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:21.978870 6647 hybrid_clock.cc:648] HybridClock initialized: now 1790280501978849 us; error 31 us; skew 500 ppm
Sep 24 20:08:21 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:21.981792 6647 init.cc:377] Logged in from keytab as oryx/127.0.115.129@KRBTEST.COM (short username oryx)
I20260924 20:08:21.982635 6647 webserver.cc:460] Webserver started at http://127.0.115.129:45597/ using document root <none> and password file <none>
I20260924 20:08:21.983201 6647 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:21.983292 6647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:21.983520 6647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:21.985617 6647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "69a0ac3b29d3410baa2befb435002333"
format_stamp: "Formatted at 2026-09-24 20:08:21 on dist-test-slave-2pm7"
server_key: "87111458e390e86a5d566aeef51bc01c"
server_key_iv: "a7aac26a70d0b707af7d5aee212b7650"
server_key_version: "encryptionkey@0"
I20260924 20:08:21.986218 6647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "69a0ac3b29d3410baa2befb435002333"
format_stamp: "Formatted at 2026-09-24 20:08:21 on dist-test-slave-2pm7"
server_key: "87111458e390e86a5d566aeef51bc01c"
server_key_iv: "a7aac26a70d0b707af7d5aee212b7650"
server_key_version: "encryptionkey@0"
I20260924 20:08:21.990083 6647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:21.992560 6662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:21.993806 6647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:21.993945 6647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "69a0ac3b29d3410baa2befb435002333"
format_stamp: "Formatted at 2026-09-24 20:08:21 on dist-test-slave-2pm7"
server_key: "87111458e390e86a5d566aeef51bc01c"
server_key_iv: "a7aac26a70d0b707af7d5aee212b7650"
server_key_version: "encryptionkey@0"
I20260924 20:08:21.994067 6647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:22.006878 6647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:22.010329 6647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:22.010560 6647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:22.011322 6647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:22.012352 6647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:22.012404 6647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.012447 6647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:22.012468 6647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.021977 6647 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:44821
I20260924 20:08:22.022001 6775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:44821 every 8 connection(s)
I20260924 20:08:22.022981 6647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:22.029233 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6647
I20260924 20:08:22.029381 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:22.029747 462 external_mini_cluster.cc:1428] Setting key ad3b3e72c9bac240777c40c4df31ea36
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:22.037273 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.024896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:51503 (local address 127.0.115.190:44475)
0924 20:08:22.025059 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.025063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.025939 (+ 876us) server_negotiation.cc:407] Connection header received
0924 20:08:22.027238 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.027242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.027288 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.027360 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.028624 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.029180 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.030666 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.030926 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.031075 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.033605 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.033613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.033615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.033640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.035141 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.035652 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.035657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.035659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.035742 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.036138 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.036142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.036144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.036337 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.036446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.036955 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.037087 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:08:22.038116 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.025270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:51503)
0924 20:08:22.025778 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.025823 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.027033 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.027530 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.027537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.027888 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.028488 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.028502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.029408 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.029413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.030561 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.030574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.031125 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.031129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.031334 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.031964 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.031999 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.033426 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.035266 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.035273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.035289 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.035549 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.035836 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.035840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.035842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.036038 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.036446 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.036454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.036708 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.036899 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.037230 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260924 20:08:22.039407 6776 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44475
I20260924 20:08:22.039705 6776 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:08:22.040395 6776 heartbeater.cc:511] Master 127.0.115.190:44475 requested a full tablet report, sending...
I20260924 20:08:22.042021 6585 ts_manager.cc:194] Registered new tserver with Master: 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.130@KRBTEST.COM" created.
I20260924 20:08:22.043264 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:51503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab.
Entry for principal oryx/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab.
I20260924 20:08:22.058403 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44475
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab
--principal=oryx/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:22.172051 6781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:22.172348 6781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:22.172390 6781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:22.176595 6781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:22.176668 6781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:22.176777 6781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:22.182137 6781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab
--principal=oryx/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:22.183248 6781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:22.184401 6781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:22.194157 6789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.194175 6787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:22.194422 6781 server_base.cc:1034] running on GCE node
W20260924 20:08:22.194427 6786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:22.194936 6781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:22.195567 6781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:22.196758 6781 hybrid_clock.cc:648] HybridClock initialized: now 1790280502196742 us; error 30 us; skew 500 ppm
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:22.199693 6781 init.cc:377] Logged in from keytab as oryx/127.0.115.130@KRBTEST.COM (short username oryx)
I20260924 20:08:22.200557 6781 webserver.cc:460] Webserver started at http://127.0.115.130:35687/ using document root <none> and password file <none>
I20260924 20:08:22.201067 6781 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:22.201123 6781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:22.201292 6781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:22.203235 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "8fc75e3aa684468382372fa979d43f85"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "df8783f5c5b09fad48414a995370fc66"
server_key_iv: "24804b439207f7d60982490058204832"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.203775 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "8fc75e3aa684468382372fa979d43f85"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "df8783f5c5b09fad48414a995370fc66"
server_key_iv: "24804b439207f7d60982490058204832"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.207603 6781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 20:08:22.210050 6796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.211364 6781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:08:22.211454 6781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "8fc75e3aa684468382372fa979d43f85"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "df8783f5c5b09fad48414a995370fc66"
server_key_iv: "24804b439207f7d60982490058204832"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.211537 6781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:22.224149 6781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:22.227526 6781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:22.227677 6781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:22.228305 6781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:22.229091 6781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:22.229136 6781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.229175 6781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:22.229190 6781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.238435 6781 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:44049
I20260924 20:08:22.238518 6909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:44049 every 8 connection(s)
I20260924 20:08:22.239377 6781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:22.245214 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6781
I20260924 20:08:22.245328 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:22.245649 462 external_mini_cluster.cc:1428] Setting key f5ada9dfef9ab587626b60b3795ad64c
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:22.253321 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.241242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35603 (local address 127.0.115.190:44475)
0924 20:08:22.241381 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.241385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.242261 (+ 876us) server_negotiation.cc:407] Connection header received
0924 20:08:22.243608 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.243612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.243667 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.243750 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.244940 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.245790 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.246924 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.247155 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.247273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.249637 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.249648 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.249651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.249673 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.251187 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.251702 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.251707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.251709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.251791 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.252200 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.252205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.252207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.252396 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.252503 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.253018 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.253143 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:08:22.254181 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.241624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:35603)
0924 20:08:22.242105 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.242157 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.243419 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.243926 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.243933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.244342 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.244813 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.244828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.245911 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.245914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.246789 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.246798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.247303 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.247307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.247477 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.248089 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.248104 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.249475 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.251318 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.251323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.251338 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.251596 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.251892 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.251895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.251897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.252104 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.252503 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.252511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.252775 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.252960 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.253283 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":143,"threads_started":1}
WARNING: no policy specified for oryx/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:08:22.255350 6910 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44475
I20260924 20:08:22.255635 6910 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:22.256179 6910 heartbeater.cc:511] Master 127.0.115.190:44475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.131@KRBTEST.COM" created.
I20260924 20:08:22.257328 6585 ts_manager.cc:194] Registered new tserver with Master: 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049)
I20260924 20:08:22.258248 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:35603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab.
Entry for principal oryx/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab.
I20260924 20:08:22.274641 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44475
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab
--principal=oryx/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:22.387375 6915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:22.387648 6915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:22.387743 6915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:22.391849 6915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:22.391911 6915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:22.392037 6915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:22.397294 6915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab
--principal=oryx/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:22.398417 6915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:22.399489 6915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:22.409364 6921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.409433 6923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.409409 6920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:22.409672 6915 server_base.cc:1034] running on GCE node
I20260924 20:08:22.410086 6915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:22.410821 6915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:22.412016 6915 hybrid_clock.cc:648] HybridClock initialized: now 1790280502411977 us; error 57 us; skew 500 ppm
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:22.414785 6915 init.cc:377] Logged in from keytab as oryx/127.0.115.131@KRBTEST.COM (short username oryx)
I20260924 20:08:22.415679 6915 webserver.cc:460] Webserver started at http://127.0.115.131:44499/ using document root <none> and password file <none>
I20260924 20:08:22.416270 6915 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:22.416332 6915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:22.416496 6915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:22.418475 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "720ffa94430342fb91f1b1c247992fce"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "67f20b74179de4c04e0eead2cd82ad97"
server_key_iv: "ce009f56e0e979038ce98fa5c7ee1c3f"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.419003 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "720ffa94430342fb91f1b1c247992fce"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "67f20b74179de4c04e0eead2cd82ad97"
server_key_iv: "ce009f56e0e979038ce98fa5c7ee1c3f"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.422813 6915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:08:22.425331 6930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.426606 6915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:22.426705 6915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "720ffa94430342fb91f1b1c247992fce"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "67f20b74179de4c04e0eead2cd82ad97"
server_key_iv: "ce009f56e0e979038ce98fa5c7ee1c3f"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.426807 6915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:22.439679 6915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:22.443089 6915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:22.443248 6915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:22.443801 6915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:22.444658 6915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:22.444703 6915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.444744 6915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:22.444768 6915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.453606 6915 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:43971
I20260924 20:08:22.453622 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:43971 every 8 connection(s)
I20260924 20:08:22.454591 6915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:22.460958 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 6915
I20260924 20:08:22.461103 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:22.461452 462 external_mini_cluster.cc:1428] Setting key 4dd8215e3db7ceea6424c0f8e7a887bd
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:22.469034 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.456566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:51845 (local address 127.0.115.190:44475)
0924 20:08:22.456719 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.456723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.457565 (+ 842us) server_negotiation.cc:407] Connection header received
0924 20:08:22.458945 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.458948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.459004 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.459115 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.460369 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.460969 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.462590 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.462861 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.462979 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.465629 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.465632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.465634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.465662 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.466877 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.467403 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.467407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.467409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.467491 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.467851 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.467855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.467857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.468105 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.468222 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.468747 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.468870 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:08:22.469885 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.456963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:51845)
0924 20:08:22.457416 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.457459 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.458729 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.459300 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.459308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.459676 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.460224 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.460238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.461101 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.461105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.462481 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.462493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.463031 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.463035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.463233 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.463859 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.463875 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.465452 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.467000 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.467006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.467022 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.467294 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.467586 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.467590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.467592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.467748 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.468221 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.468229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.468487 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.468704 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.469046 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260924 20:08:22.471014 7044 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44475
I20260924 20:08:22.471239 7044 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:22.471706 7044 heartbeater.cc:511] Master 127.0.115.190:44475 requested a full tablet report, sending...
I20260924 20:08:22.472764 6585 ts_manager.cc:194] Registered new tserver with Master: 720ffa94430342fb91f1b1c247992fce (127.0.115.131:43971)
I20260924 20:08:22.473304 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:51845
I20260924 20:08:22.475607 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.115.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.189.keytab.
Entry for principal oryx/127.0.115.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.189.keytab.
I20260924 20:08:22.503099 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:38309
--webserver_interface=127.0.115.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.189.keytab
--principal=oryx/127.0.115.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44475,127.0.115.189:38309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:22.618832 7049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:22.619138 7049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:22.619221 7049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:22.623479 7049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:22.623553 7049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:22.623574 7049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:22.623595 7049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:22.623611 7049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:22.629218 7049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44475,127.0.115.189:38309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.189.keytab
--principal=oryx/127.0.115.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:38309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.115.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
W20260924 20:08:22.629469 7049 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.115.190:44475,127.0.115.189:38309'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 20:08:22.630517 7049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:22.631685 7049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:22.640654 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.640854 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.641029 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:22.641018 7049 server_base.cc:1034] running on GCE node
I20260924 20:08:22.641801 7049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:22.642927 7049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:22.644086 7049 hybrid_clock.cc:648] HybridClock initialized: now 1790280502644069 us; error 27 us; skew 500 ppm
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:22.646905 7049 init.cc:377] Logged in from keytab as oryx/127.0.115.189@KRBTEST.COM (short username oryx)
I20260924 20:08:22.647841 7049 webserver.cc:460] Webserver started at http://127.0.115.189:45775/ using document root <none> and password file <none>
I20260924 20:08:22.648447 7049 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:22.648538 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:22.648777 7049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:22.650827 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/instance:
uuid: "da565a0436144a6895b0bf8e0dbb1603"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "275df0e9e6bd700f303b518e1522fee6"
server_key_iv: "1f80c12208998a87429703f5d310ef76"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.651399 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal/instance:
uuid: "da565a0436144a6895b0bf8e0dbb1603"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "275df0e9e6bd700f303b518e1522fee6"
server_key_iv: "1f80c12208998a87429703f5d310ef76"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.655179 7049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:08:22.657765 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.659008 7049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:22.659152 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
uuid: "da565a0436144a6895b0bf8e0dbb1603"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "275df0e9e6bd700f303b518e1522fee6"
server_key_iv: "1f80c12208998a87429703f5d310ef76"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.659268 7049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:22.686164 7049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:22.689693 7049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:22.689917 7049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:22.697310 7049 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.189:38309
I20260924 20:08:22.697666 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.189:38309 every 8 connection(s)
I20260924 20:08:22.698340 7049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
I20260924 20:08:22.700335 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 7049
I20260924 20:08:22.701826 7117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:22.706331 7117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.115.189@KRBTEST.COM
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.189@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:22.715950 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.701824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.0.115.189:38309)
0924 20:08:22.702260 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.702278 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.702313 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:08:22.702976 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.702993 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.703320 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.703636 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.704533 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.705776 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.706904 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.707368 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.707484 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.710425 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.710431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.710443 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.710473 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.712458 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.712969 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.712974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.712978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.713084 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.713380 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.713382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.713383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.713751 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.713976 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.714219 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.714561 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":123,"threads_started":1}
I20260924 20:08:22.718248 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.708361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.0.115.190:44475)
0924 20:08:22.708523 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.708530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.708891 (+ 361us) server_negotiation.cc:407] Connection header received
0924 20:08:22.709022 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.709026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.709078 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.709160 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.709886 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.710518 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.711866 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.712251 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.712369 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.714748 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.714752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.714755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.714790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.716426 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.716896 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.716901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.716903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.716995 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.717381 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.717383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.717384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.717597 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.717676 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.718039 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.718135 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:08:22.718400 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.708320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:56516)
0924 20:08:22.708751 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.708801 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.708917 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.709217 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.709222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.709611 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.709760 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.709767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.710682 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.710685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.711742 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.711753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.712363 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.712366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.712450 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.712970 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.712985 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.714612 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.716538 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.716546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.716551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.716791 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.717111 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.717167 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.717171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.717275 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.717707 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.717712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.717829 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.717989 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.718178 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"spinlock_wait_cycles":896,"thread_start_us":179,"threads_started":1}
I20260924 20:08:22.719902 7117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.189@KRBTEST.COM for oryx/127.0.115.189@KRBTEST.COM
I20260924 20:08:22.728551 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.720451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.0.1:44096)
0924 20:08:22.720682 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.720700 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.720817 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.721054 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.721056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.721234 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.721315 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.721320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.722423 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.722426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.722941 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.722948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.723414 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.723418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.723542 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.724042 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.724053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.725343 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.727173 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.727175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.727177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.727380 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.727623 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.727625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.727626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.727693 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.728114 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.728116 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.728193 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.728289 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.728402 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1}
I20260924 20:08:22.728574 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.720524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.0.115.189:38309)
0924 20:08:22.720683 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.720688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.720773 (+ 85us) server_negotiation.cc:407] Connection header received
0924 20:08:22.720911 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.720914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.720952 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.721045 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.721423 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.722297 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.723055 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.723307 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.723417 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.725472 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.725475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.725477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.725498 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.726992 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.727468 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.727471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.727472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.727541 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.727770 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.727773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.727774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.728014 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.728093 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.728336 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.728451 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":75}
I20260924 20:08:22.732163 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Bootstrap starting.
I20260924 20:08:22.734515 7117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:22.735347 7117 log.cc:824] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:22.737597 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: No bootstrap required, opened a new log
I20260924 20:08:22.741710 7117 raft_consensus.cc:348] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } }
I20260924 20:08:22.742013 7117 raft_consensus.cc:374] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:22.742062 7117 raft_consensus.cc:729] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da565a0436144a6895b0bf8e0dbb1603, State: Initialized, Role: FOLLOWER
I20260924 20:08:22.742543 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } }
I20260924 20:08:22.743008 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } } }
I20260924 20:08:22.743151 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:08:22.743860 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:08:22.749056 7134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:08:22.749202 7134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:08:22.751071 7049 master_runner.cc:428] Detected that this master da565a0436144a6895b0bf8e0dbb1603 is joining an existing cluster
I20260924 20:08:22.751134 7049 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.115.189:38309
I20260924 20:08:22.764241 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.753692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.0.115.189:38309)
0924 20:08:22.753831 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.753836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.754053 (+ 217us) server_negotiation.cc:407] Connection header received
0924 20:08:22.754209 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.754214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.754269 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.754346 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.755243 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.756066 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.757271 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.757498 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.757616 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.759719 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.759723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.759725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.759747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.761869 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.762703 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.762706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.762708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.762802 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.763086 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.763090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.763091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.763359 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.763428 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.763920 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.764066 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:08:22.764810 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.753567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.0.1:44104)
0924 20:08:22.753980 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.753994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.754116 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.754459 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.754461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.754780 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.755115 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.755125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.756209 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.756213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.757156 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.757165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.757619 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.757623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.757708 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.759123 (+ 1415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.759142 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.759579 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.761992 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.761998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.762001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.762591 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.762900 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.762902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.762904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.762989 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.763671 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.763673 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.763769 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.764461 (+ 692us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.764597 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":85,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}
I20260924 20:08:22.765944 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.753567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:56522)
0924 20:08:22.754199 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.754212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.754302 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.754548 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.754550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.754780 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.755129 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.755135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.757101 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.757104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.758039 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.758050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.758548 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.758552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.758670 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.759182 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.759198 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.759718 (+ 520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.762013 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.762017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.762019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.762432 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.763372 (+ 940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.763376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.763380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.763466 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.764749 (+ 1283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.764751 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.764840 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.765621 (+ 781us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.765759 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":437,"spinlock_wait_cycles":7680,"thread_start_us":103,"threads_started":1}
I20260924 20:08:22.765964 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.753759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.0.115.190:44475)
0924 20:08:22.753893 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.753897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.754287 (+ 390us) server_negotiation.cc:407] Connection header received
0924 20:08:22.754391 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.754394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.754443 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.754518 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.756060 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.756971 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.758200 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.758431 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.758515 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.759859 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.759866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.759871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.759903 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.761907 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.763092 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.763096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.763098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.763247 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.763568 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.763571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.763573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.764601 (+ 1028us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.764677 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.765673 (+ 996us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.765801 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.190@KRBTEST.COM for oryx/127.0.115.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.115.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.188@KRBTEST.COM" created.
I20260924 20:08:22.781839 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.770396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.0.1:44116)
0924 20:08:22.770749 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.770800 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.770961 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.771309 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.771315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.772168 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.772337 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.772346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.773362 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.773368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.774512 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.774527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.775124 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.775131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.775274 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.775867 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.775889 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.777854 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.779880 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.779890 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.779898 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.780271 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.780534 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.780539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.780541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.780636 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.781012 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.781019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.781241 (+ 222us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.781408 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.781645 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260924 20:08:22.781996 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.770396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.0.115.189:38309)
0924 20:08:22.770516 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.770520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.770866 (+ 346us) server_negotiation.cc:407] Connection header received
0924 20:08:22.771118 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.771121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.771178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.771269 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.772478 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.773248 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.774653 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.774991 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.775097 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.778027 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.778031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.778035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.778070 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.779767 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.780359 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.780362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.780363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.780438 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.780714 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.780716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.780717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.780909 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.780981 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.781485 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.781671 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:08:22.783466 6585 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.115.189:38309: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603"
member_type: NON_VOTER
last_known_addr {
host: "127.0.115.189"
port: 38309
}
attrs {
promote: true
}
}
dest_uuid: "ab3766a2ab094b89b7abed98b7412ac0"
cas_config_opid_index: -1
I20260924 20:08:22.784281 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } }
I20260924 20:08:22.786840 7091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:22.788655 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Refusing update from remote peer ab3766a2ab094b89b7abed98b7412ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 20:08:22.789611 7147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:22.789634 7145 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } }
I20260924 20:08:22.791899 7145 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.115.189:38309
I20260924 20:08:22.792431 7049 master.cc:537] Master@127.0.115.189:38309 shutting down...
I20260924 20:08:22.794507 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } }
I20260924 20:08:22.794639 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:22.796276 7091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } }
I20260924 20:08:22.798594 7148 mvcc.cc:205] Tried to move back new op lower bound from 7332988935390982144 to 7332988935202705408. Current Snapshot: MvccSnapshot[applied={T|T < 7332988935245881344}]
I20260924 20:08:22.799365 7151 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } }
I20260924 20:08:22.799477 7151 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:22.799573 7151 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } }
I20260924 20:08:22.799644 7151 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:22.799984 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3766a2ab094b89b7abed98b7412ac0. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } }
I20260924 20:08:22.800213 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:22.800941 7049 raft_consensus.cc:2229] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Raft consensus shutting down.
I20260924 20:08:22.801285 7049 raft_consensus.cc:2258] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Raft consensus is shut down!
I20260924 20:08:22.801399 7049 tablet_replica.cc:330] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.188.keytab.
Entry for principal oryx/127.0.115.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.188.keytab.
I20260924 20:08:22.804062 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:34931
--webserver_interface=127.0.115.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.188.keytab
--principal=oryx/127.0.115.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:08:22.817914 7049 master.cc:552] Master@127.0.115.189:38309 shutdown complete.
I20260924 20:08:22.817983 7049 master_runner.cc:305] Clearing existing system tablet
I20260924 20:08:22.818249 7049 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:22.818305 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:22.819680 7049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 20:08:22.820683 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.820943 7049 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:22.821085 7049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 20:08:22.821127 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
uuid: "da565a0436144a6895b0bf8e0dbb1603"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "275df0e9e6bd700f303b518e1522fee6"
server_key_iv: "1f80c12208998a87429703f5d310ef76"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.823272 7049 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:22.825670 7049 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 20:08:22.825766 7049 log.cc:1196] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 20:08:22.826094 7049 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Deleting consensus metadata
I20260924 20:08:22.826264 7049 master_runner.cc:315] Copying system tablet from 127.0.115.190:44475
I20260924 20:08:22.827502 7049 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: tablet copy: Beginning tablet copy session from remote peer at address 127.0.115.190:44475
I20260924 20:08:22.837769 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.828221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.0.115.190:44475)
0924 20:08:22.828365 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:22.828370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:22.829428 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 20:08:22.829551 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:22.829555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:22.829614 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:22.829716 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:22.830287 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.831144 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.832376 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.832694 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.832802 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.834151 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:22.834155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:22.834157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:22.834185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:22.835873 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.836410 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.836413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.836416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.836513 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:22.836811 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:22.836814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:22.836816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:22.837043 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:22.837156 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:22.837483 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 20:08:22.837611 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:08:22.838182 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:22.828115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:56530)
0924 20:08:22.829309 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:22.829324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:22.829451 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:22.829764 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:22.829767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:22.829989 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:22.830113 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.830120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.831282 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.831288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.832231 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:22.832242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:22.832822 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:22.832826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:22.832927 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:22.833494 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:22.833510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:22.833970 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:22.836036 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.836039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.836042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.836307 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.836614 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:22.836617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:22.836619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:22.836710 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:22.837162 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:22.837164 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:22.837290 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 20:08:22.837400 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 20:08:22.837549 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":84,"threads_started":1}
I20260924 20:08:22.838850 6605 tablet_copy_service.cc:140] P ab3766a2ab094b89b7abed98b7412ac0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer da565a0436144a6895b0bf8e0dbb1603 ({username='oryx', principal='oryx/127.0.115.189@KRBTEST.COM'} at 127.0.0.1:56530)
I20260924 20:08:22.839043 6605 tablet_copy_service.cc:161] P ab3766a2ab094b89b7abed98b7412ac0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer da565a0436144a6895b0bf8e0dbb1603 at {username='oryx', principal='oryx/127.0.115.189@KRBTEST.COM'} at 127.0.0.1:56530: session id = da565a0436144a6895b0bf8e0dbb1603-00000000000000000000000000000000
I20260924 20:08:22.840888 6605 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 20:08:22.842561 7049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 20:08:22.845209 7049 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: tablet copy: Starting download of 0 data blocks...
I20260924 20:08:22.845418 7049 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: tablet copy: Starting download of 1 WAL segments...
I20260924 20:08:22.847266 7049 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 20:08:22.849123 6605 tablet_copy_service.cc:342] P ab3766a2ab094b89b7abed98b7412ac0: Request end of tablet copy session da565a0436144a6895b0bf8e0dbb1603-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.115.189@KRBTEST.COM'} at 127.0.0.1:56530
I20260924 20:08:22.849249 6605 tablet_copy_service.cc:433] P ab3766a2ab094b89b7abed98b7412ac0: ending tablet copy session da565a0436144a6895b0bf8e0dbb1603-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer da565a0436144a6895b0bf8e0dbb1603
I20260924 20:08:22.857391 7049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:22.859148 7165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.859184 7166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.859587 7168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:22.860303 7049 server_base.cc:1034] running on GCE node
I20260924 20:08:22.860503 7049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:22.860908 7049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:22.862118 7049 hybrid_clock.cc:648] HybridClock initialized: now 1790280502862103 us; error 31 us; skew 500 ppm
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:22.863919 7049 init.cc:377] Logged in from keytab as oryx/127.0.115.189@KRBTEST.COM (short username oryx)
I20260924 20:08:22.864523 7049 webserver.cc:460] Webserver started at http://127.0.115.189:33433/ using document root <none> and password file <none>
I20260924 20:08:22.864763 7049 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:22.864856 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:22.866336 7049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:22.867399 7175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.867712 7049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:22.867789 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
uuid: "da565a0436144a6895b0bf8e0dbb1603"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "275df0e9e6bd700f303b518e1522fee6"
server_key_iv: "1f80c12208998a87429703f5d310ef76"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.867867 7049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:22.877497 7049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:22.879805 7049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:22.884917 7049 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.189:38309
I20260924 20:08:22.884949 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.189:38309 every 8 connection(s)
I20260924 20:08:22.885666 7049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
I20260924 20:08:22.887537 7228 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:08:22.888715 7228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Bootstrap starting.
I20260924 20:08:22.896617 7228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:08:22.896920 7228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: Bootstrap complete.
I20260924 20:08:22.897506 7228 raft_consensus.cc:348] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } }
I20260924 20:08:22.897650 7228 raft_consensus.cc:729] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da565a0436144a6895b0bf8e0dbb1603, State: Initialized, Role: LEARNER
I20260924 20:08:22.897758 7228 consensus_queue.cc:260] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } }
I20260924 20:08:22.898269 7228 sys_catalog.cc:564] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:22.898597 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } }
I20260924 20:08:22.898852 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:22.901708 7242 catalog_manager.cc:1259] Loaded cluster ID: 440abd657feb47f5aa8c2bb870b600ef
I20260924 20:08:22.901773 7242 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: loading cluster ID for follower catalog manager: success
I20260924 20:08:22.904086 7242 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: acquiring CA information for follower catalog manager: success
I20260924 20:08:22.905561 7242 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 20:08:22.939613 7156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:22.939913 7156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:22.940025 7156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:22.944083 7156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:22.944164 7156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:22.944201 7156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:22.944257 7156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:22.944290 7156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:22.950073 7156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.188.keytab
--principal=oryx/127.0.115.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.115.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:22.951221 7156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:22.952400 7156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:22.961860 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:22.961944 7156 server_base.cc:1034] running on GCE node
W20260924 20:08:22.961840 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:22.961992 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:22.962773 7156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:22.964028 7156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:22.965232 7156 hybrid_clock.cc:648] HybridClock initialized: now 1790280502965217 us; error 39 us; skew 500 ppm
Sep 24 20:08:22 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:22.968248 7156 init.cc:377] Logged in from keytab as oryx/127.0.115.188@KRBTEST.COM (short username oryx)
I20260924 20:08:22.969441 7156 webserver.cc:460] Webserver started at http://127.0.115.188:45023/ using document root <none> and password file <none>
I20260924 20:08:22.970080 7156 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:22.970183 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:22.970487 7156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:22.972923 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/instance:
uuid: "0f7bc2a5c89b45b5854effb879ee1512"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "1e1aaef72c470fd7f3223ecb64935c79"
server_key_iv: "624b32bc8a9be2d522c5ae2ebc400e5a"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.973623 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal/instance:
uuid: "0f7bc2a5c89b45b5854effb879ee1512"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "1e1aaef72c470fd7f3223ecb64935c79"
server_key_iv: "624b32bc8a9be2d522c5ae2ebc400e5a"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.977999 7156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:22.981201 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:22.982692 7156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:08:22.982864 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
uuid: "0f7bc2a5c89b45b5854effb879ee1512"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "1e1aaef72c470fd7f3223ecb64935c79"
server_key_iv: "624b32bc8a9be2d522c5ae2ebc400e5a"
server_key_version: "encryptionkey@0"
I20260924 20:08:22.982997 7156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:22.992776 7156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:22.995790 7156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:22.996052 7156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:23.003422 7156 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.188:34931
I20260924 20:08:23.003711 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.188:34931 every 8 connection(s)
I20260924 20:08:23.004648 7156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
I20260924 20:08:23.007537 7310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:23.011301 7310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:08:23.012037 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 7156
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.188@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.115.188@KRBTEST.COM
I20260924 20:08:23.024988 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.013456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.0.115.190:44475)
0924 20:08:23.013603 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.013606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.014268 (+ 662us) server_negotiation.cc:407] Connection header received
0924 20:08:23.015328 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.015331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.015392 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.015489 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.016510 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.017048 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.018132 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.018367 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.018446 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.021150 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.021154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.021159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.021188 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.022746 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.023324 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.023327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.023328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.023401 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.023757 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.023762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.023764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.023993 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.024102 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.024701 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.024823 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:08:23.025866 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.013693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:56546)
0924 20:08:23.014126 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.014170 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.015116 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.015555 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.015562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.016011 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.016367 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.016380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.017174 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.017178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.018011 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.018020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.018500 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.018504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.018650 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.019479 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.019495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.021008 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.022870 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.022881 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.022894 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.023217 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.023504 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.023508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.023510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.023651 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.024109 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.024117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.024430 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.024646 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.024908 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":59,"spinlock_wait_cycles":6016,"thread_start_us":100,"threads_started":1}
I20260924 20:08:23.027064 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.014375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.0.115.188:34931)
0924 20:08:23.014623 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.014633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.014664 (+ 31us) server_negotiation.cc:407] Connection header received
0924 20:08:23.015125 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.015147 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.015336 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.015516 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.016367 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.017349 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.018158 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.018576 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.018765 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.021544 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.021558 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.021568 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.021589 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.024170 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.025224 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.025230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.025231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.025322 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.025635 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.025638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.025640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.025915 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.026720 (+ 805us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.026826 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.026936 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260924 20:08:23.028002 7310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.188@KRBTEST.COM for oryx/127.0.115.189@KRBTEST.COM
I20260924 20:08:23.036525 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.028485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.0.1:44132)
0924 20:08:23.028818 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.028832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.028909 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.029147 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.029151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.029301 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.029387 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.029392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.030242 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.030244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.030831 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.030839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.031280 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.031283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.031466 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.031869 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.031880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.033145 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.034994 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.034998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.035001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.035194 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.035491 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.035494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.035496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.035580 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.036015 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.036018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.036112 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.036220 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.036353 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260924 20:08:23.036553 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.028586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.0.115.189:38309)
0924 20:08:23.028926 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.028929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.028961 (+ 32us) server_negotiation.cc:407] Connection header received
0924 20:08:23.028998 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.029000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.029039 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.029103 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.029488 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.030132 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.030928 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.031150 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.031281 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.033289 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.033293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.033300 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.033323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.034847 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.035296 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.035300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.035302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.035397 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.035671 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.035677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.035682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.035864 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.035992 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.036284 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.036400 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":151,"threads_started":1}
I20260924 20:08:23.037366 7310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280502, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.188@KRBTEST.COM for oryx/127.0.115.188@KRBTEST.COM
I20260924 20:08:23.044389 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.037836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.0.1:39944)
0924 20:08:23.037947 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.037963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.038036 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.038208 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.038210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.038333 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.038401 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.038405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.039064 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.039067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.039554 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.039561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.040014 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.040018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.040087 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.040504 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.040524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.041688 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.043077 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.043083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.043088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.043322 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.043542 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.043544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.043546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.043616 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.043922 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.043924 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.044057 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.044140 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.044233 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 20:08:23.044405 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.037923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.0.115.188:34931)
0924 20:08:23.038032 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.038042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.038055 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:23.038084 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.038086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.038117 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.038178 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.038492 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.038973 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.039646 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.039876 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.040000 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.041802 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.041805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.041807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.041825 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.042971 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.043403 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.043405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.043406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.043469 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.043690 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.043692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.043693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.043847 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.043917 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.044203 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.044288 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:08:23.045868 6776 heartbeater.cc:503] Master 127.0.115.190:44475 was elected leader, sending a full tablet report...
I20260924 20:08:23.047840 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Bootstrap starting.
I20260924 20:08:23.050222 7310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:23.050966 7310 log.cc:824] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:23.052982 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: No bootstrap required, opened a new log
I20260924 20:08:23.056061 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } }
I20260924 20:08:23.056263 7310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:23.056335 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7bc2a5c89b45b5854effb879ee1512, State: Initialized, Role: FOLLOWER
I20260924 20:08:23.056751 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } }
I20260924 20:08:23.057260 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } } }
I20260924 20:08:23.057380 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:08:23.057981 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:08:23.063205 7327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:08:23.063308 7327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:08:23.066504 7156 master_runner.cc:428] Detected that this master 0f7bc2a5c89b45b5854effb879ee1512 is joining an existing cluster
I20260924 20:08:23.066576 7156 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.115.188:34931
I20260924 20:08:23.078773 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.069675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.0.115.188:34931)
0924 20:08:23.069776 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.069780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.070315 (+ 535us) server_negotiation.cc:407] Connection header received
0924 20:08:23.070445 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.070450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.070487 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.070535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.071086 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.071796 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.073495 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.074130 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.074189 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.075508 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.075512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.075515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.075537 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.076980 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.077486 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.077489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.077491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.077562 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.077874 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.077878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.077880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.078079 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.078173 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.078446 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.078591 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":44}
I20260924 20:08:23.078996 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.069067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.0.115.190:44475)
0924 20:08:23.069206 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.069211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.069627 (+ 416us) server_negotiation.cc:407] Connection header received
0924 20:08:23.069820 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.069823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.069876 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.069930 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.070639 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.071485 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.072738 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.073088 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.073165 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.074694 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.074697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.074700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.074727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.076915 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.077460 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.077465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.077468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.077564 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.078012 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.078015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.078017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.078261 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.078361 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.078690 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.078820 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:08:23.078801 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.069615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.0.1:39950)
0924 20:08:23.070251 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.070264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.070362 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.070668 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.070670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.070842 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.070974 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.070980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.071908 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.071911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.073390 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.073399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.074348 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.074351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.074416 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.074868 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.074879 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.075321 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.077095 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.077101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.077104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.077361 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.077668 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.077674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.077679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.077783 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.078182 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.078185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.078270 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.078381 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.078527 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":50,"thread_start_us":88,"threads_started":1}
I20260924 20:08:23.079495 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.069020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:56552)
0924 20:08:23.069539 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.069555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.069702 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.070082 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.070085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.070336 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.070498 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.070505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.071625 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.071628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.072598 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.072609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.073348 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.073354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.073448 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.074006 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.074026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.074542 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.077037 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.077040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.077042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.077348 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.077863 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.077865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.077866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.077920 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.078405 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.078408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.078486 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.079140 (+ 654us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.079280 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"spinlock_wait_cycles":12160,"thread_start_us":265,"threads_started":1}
I20260924 20:08:23.080538 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6647
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280501, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.190@KRBTEST.COM for oryx/127.0.115.188@KRBTEST.COM
I20260924 20:08:23.089773 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:44821
--local_ip_for_outbound_sockets=127.0.115.129
--tserver_master_addrs=127.0.115.190:44475
--webserver_port=45597
--webserver_interface=127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab
--principal=oryx/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:08:23.099788 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.083859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.0.1:39964)
0924 20:08:23.083981 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.083996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.084072 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.084373 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.084379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.084589 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.084728 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.084738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.085719 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.085725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.086748 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.086762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.087503 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.087527 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.087627 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.088077 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.088092 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.089660 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.098052 (+ 8392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.098056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.098059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.098325 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.098674 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.098677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.098679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.098770 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.099246 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.099250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.099363 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.099452 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.099608 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 20:08:23.100764 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.083976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.0.115.188:34931)
0924 20:08:23.084073 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.084078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.084090 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:08:23.084188 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.084191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.084240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.084318 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.084850 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.085598 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.086880 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.087239 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.087338 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.089792 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.089796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.089798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.089822 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.097909 (+ 8087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.098447 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.098450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.098452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.098527 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.098880 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.098883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.098884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.099102 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.100392 (+ 1290us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.100473 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.100615 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 20:08:23.101943 6585 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.115.188:34931: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512"
member_type: NON_VOTER
last_known_addr {
host: "127.0.115.188"
port: 34931
}
attrs {
promote: true
}
}
dest_uuid: "ab3766a2ab094b89b7abed98b7412ac0"
cas_config_opid_index: 5
I20260924 20:08:23.102234 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.069128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.0.115.189:38309)
0924 20:08:23.069260 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.069265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.069990 (+ 725us) server_negotiation.cc:407] Connection header received
0924 20:08:23.070118 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.070121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.070173 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.070246 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.070940 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.071776 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.073151 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.073525 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.073620 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.098041 (+ 24421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.098048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.098053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.098093 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.100177 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.100863 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.100869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.100874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.100969 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.101338 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.101341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.101343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.101561 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.101652 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.101958 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.102085 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 20:08:23.102581 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } }
I20260924 20:08:23.103026 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.069024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.0.1:44148)
0924 20:08:23.069903 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.069913 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.070025 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.070371 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.070374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.070666 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.070795 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.070802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.071908 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.071911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.073021 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.073031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.073776 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.073780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.073891 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.096746 (+ 22855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.096768 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.097789 (+ 1021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.100377 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.100382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.100386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.100727 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.101102 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.101106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.101108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.101209 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.101696 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.101699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.101787 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.102648 (+ 861us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.102856 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":144,"thread_start_us":116,"threads_started":1}
I20260924 20:08:23.103892 7145 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.105026 7284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:23.105352 7145 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.115.188:34931
I20260924 20:08:23.105517 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.105607 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:23.105973 7156 master.cc:537] Master@127.0.115.188:34931 shutting down...
I20260924 20:08:23.106923 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Refusing update from remote peer ab3766a2ab094b89b7abed98b7412ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 20:08:23.108630 7147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:08:23.110455 6572 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 20:08:23.112040 6572 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 -> Peer 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188:34931): Couldn't send request to peer 0f7bc2a5c89b45b5854effb879ee1512. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 20:08:23.113528 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.104089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.0.1:44156)
0924 20:08:23.108112 (+ 4023us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.108127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.108204 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.109239 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.109242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.109271 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:23.109358 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.109363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.110402 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.110405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.111657 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.111671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.113081 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.113084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.113170 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.113214 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.113307 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.113391 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3941}
I20260924 20:08:23.113576 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.104307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.0.115.189:38309)
0924 20:08:23.108894 (+ 4587us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.108899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.108911 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:08:23.108951 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.108971 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.109027 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.109115 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:23.109483 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.110276 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.111855 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.112752 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.112860 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.113016 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.113346 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.113433 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4514}
I20260924 20:08:23.113917 7156 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:23.114100 7202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Refusing update from remote peer ab3766a2ab094b89b7abed98b7412ac0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 20:08:23.114279 7156 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:23.114405 7156 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: stopping tablet replica
I20260924 20:08:23.114501 7147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 20:08:23.115814 7202 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.117595 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3766a2ab094b89b7abed98b7412ac0. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.117717 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:23.117805 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: NON_VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: true } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.117877 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:23.118351 7145 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: attempting to promote NON_VOTER da565a0436144a6895b0bf8e0dbb1603 to VOTER
I20260924 20:08:23.118786 7145 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } }
I20260924 20:08:23.119294 7156 master.cc:552] Master@127.0.115.188:34931 shutdown complete.
I20260924 20:08:23.119331 7156 master_runner.cc:305] Clearing existing system tablet
I20260924 20:08:23.119566 7156 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:23.119621 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:23.120252 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.119971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.0.1:39968)
0924 20:08:23.120114 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.120180 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:34931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 20:08:23.120807 7202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 LEARNER]: Refusing update from remote peer ab3766a2ab094b89b7abed98b7412ac0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 20:08:23.121160 7147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 20:08:23.121690 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.121361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.0.1:39982)
0924 20:08:23.121488 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.121532 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:34931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 20:08:23.121819 6572 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 -> Peer 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188:34931): Couldn't send request to peer 0f7bc2a5c89b45b5854effb879ee1512. Status: Network error: Client connection negotiation failed: client connection to 127.0.115.188:34931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 20:08:23.122191 7156 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:23.122833 7147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.124248 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.124361 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:23.124475 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.124547 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:23.125049 7202 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.126505 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.126617 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:08:23.128170 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.128386 7156 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:23.128522 7156 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 20:08:23.128567 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
uuid: "0f7bc2a5c89b45b5854effb879ee1512"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "1e1aaef72c470fd7f3223ecb64935c79"
server_key_iv: "624b32bc8a9be2d522c5ae2ebc400e5a"
server_key_version: "encryptionkey@0"
I20260924 20:08:23.130596 7156 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:23.132877 7156 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 20:08:23.132968 7156 log.cc:1196] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 20:08:23.133210 7156 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Deleting consensus metadata
I20260924 20:08:23.133384 7156 master_runner.cc:315] Copying system tablet from 127.0.115.190:44475
I20260924 20:08:23.134547 7156 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.115.190:44475
I20260924 20:08:23.144567 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.135181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.0.115.190:44475)
0924 20:08:23.135312 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.135317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.136296 (+ 979us) server_negotiation.cc:407] Connection header received
0924 20:08:23.136431 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.136435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.136494 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.136584 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.137123 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.137959 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.139133 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.139479 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.139586 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.140952 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.140956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.140959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.140986 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.142745 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.143237 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.143241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.143243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.143335 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.143626 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.143629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.143632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.143845 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.143989 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.144272 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.144392 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:08:23.144953 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.135087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:56554)
0924 20:08:23.136182 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.136197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.136327 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.136619 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.136622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.136868 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.136990 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.136998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.138084 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.138088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.138999 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.139009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.139606 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.139610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.139702 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.140321 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.140337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.140792 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.142867 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.142871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.142873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.143136 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.143433 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.143436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.143438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.143532 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.143975 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.143978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.144084 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.144195 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.144438 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":89,"threads_started":1}
I20260924 20:08:23.145419 6605 tablet_copy_service.cc:140] P ab3766a2ab094b89b7abed98b7412ac0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0f7bc2a5c89b45b5854effb879ee1512 ({username='oryx', principal='oryx/127.0.115.188@KRBTEST.COM'} at 127.0.0.1:56554)
I20260924 20:08:23.145524 6605 tablet_copy_service.cc:161] P ab3766a2ab094b89b7abed98b7412ac0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0f7bc2a5c89b45b5854effb879ee1512 at {username='oryx', principal='oryx/127.0.115.188@KRBTEST.COM'} at 127.0.0.1:56554: session id = 0f7bc2a5c89b45b5854effb879ee1512-00000000000000000000000000000000
I20260924 20:08:23.146601 6605 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 20:08:23.148006 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 20:08:23.150717 7156 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: tablet copy: Starting download of 0 data blocks...
I20260924 20:08:23.150913 7156 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: tablet copy: Starting download of 1 WAL segments...
I20260924 20:08:23.152482 7156 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 20:08:23.154273 6605 tablet_copy_service.cc:342] P ab3766a2ab094b89b7abed98b7412ac0: Request end of tablet copy session 0f7bc2a5c89b45b5854effb879ee1512-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.115.188@KRBTEST.COM'} at 127.0.0.1:56554
I20260924 20:08:23.154362 6605 tablet_copy_service.cc:433] P ab3766a2ab094b89b7abed98b7412ac0: ending tablet copy session 0f7bc2a5c89b45b5854effb879ee1512-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0f7bc2a5c89b45b5854effb879ee1512
I20260924 20:08:23.162178 7156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:23.164032 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:23.164049 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:23.164294 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:23.164497 7156 server_base.cc:1034] running on GCE node
I20260924 20:08:23.164824 7156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:23.165102 7156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:23.166249 7156 hybrid_clock.cc:648] HybridClock initialized: now 1790280503166236 us; error 29 us; skew 500 ppm
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:23.168075 7156 init.cc:377] Logged in from keytab as oryx/127.0.115.188@KRBTEST.COM (short username oryx)
I20260924 20:08:23.168581 7156 webserver.cc:460] Webserver started at http://127.0.115.188:44793/ using document root <none> and password file <none>
I20260924 20:08:23.168823 7156 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:23.168912 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:23.170439 7156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:23.171456 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.171748 7156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:23.171821 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
uuid: "0f7bc2a5c89b45b5854effb879ee1512"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "1e1aaef72c470fd7f3223ecb64935c79"
server_key_iv: "624b32bc8a9be2d522c5ae2ebc400e5a"
server_key_version: "encryptionkey@0"
I20260924 20:08:23.171895 7156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:23.199681 7156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:23.202119 7156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:23.207736 7156 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.188:34931
I20260924 20:08:23.207777 7410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.188:34931 every 8 connection(s)
I20260924 20:08:23.208374 7156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
I20260924 20:08:23.214058 7411 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:08:23.215181 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Bootstrap starting.
I20260924 20:08:23.227155 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:08:23.227509 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Bootstrap complete.
I20260924 20:08:23.228145 7411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } }
I20260924 20:08:23.228302 7411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0f7bc2a5c89b45b5854effb879ee1512, State: Initialized, Role: LEARNER
I20260924 20:08:23.228420 7411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } }
I20260924 20:08:23.228803 7414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.228914 7411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:23.229027 7414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:23.232309 7425 catalog_manager.cc:1259] Loaded cluster ID: 440abd657feb47f5aa8c2bb870b600ef
I20260924 20:08:23.232426 7425 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: loading cluster ID for follower catalog manager: success
I20260924 20:08:23.234918 7425 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: acquiring CA information for follower catalog manager: success
I20260924 20:08:23.236389 7425 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 20:08:23.242491 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:23.242769 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:23.242874 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:23.247135 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:23.247216 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:23.247304 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:23.252799 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab
--principal=oryx/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:44821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=45597
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:23.253937 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:23.255007 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:23.260834 6910 heartbeater.cc:503] Master 127.0.115.190:44475 was elected leader, sending a full tablet report...
W20260924 20:08:23.265174 7433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:23.265244 7431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:23.265201 7430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:23.265429 7336 server_base.cc:1034] running on GCE node
I20260924 20:08:23.265954 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:23.266695 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:23.267884 7336 hybrid_clock.cc:648] HybridClock initialized: now 1790280503267866 us; error 27 us; skew 500 ppm
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:23.270728 7336 init.cc:377] Logged in from keytab as oryx/127.0.115.129@KRBTEST.COM (short username oryx)
I20260924 20:08:23.271623 7336 webserver.cc:460] Webserver started at http://127.0.115.129:45597/ using document root <none> and password file <none>
I20260924 20:08:23.272284 7336 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:23.272361 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:23.276669 7336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:08:23.279204 7440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.280839 7336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:23.281018 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "69a0ac3b29d3410baa2befb435002333"
format_stamp: "Formatted at 2026-09-24 20:08:21 on dist-test-slave-2pm7"
server_key: "87111458e390e86a5d566aeef51bc01c"
server_key_iv: "a7aac26a70d0b707af7d5aee212b7650"
server_key_version: "encryptionkey@0"
I20260924 20:08:23.281893 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:23.296748 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:23.300344 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:23.300551 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:23.301172 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:23.302054 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:23.302125 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.302193 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:23.302235 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.312028 7336 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:44821
I20260924 20:08:23.312083 7553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:44821 every 8 connection(s)
I20260924 20:08:23.313107 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:23.317165 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 7336
I20260924 20:08:23.317359 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6781
I20260924 20:08:23.325744 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:44049
--local_ip_for_outbound_sockets=127.0.115.130
--tserver_master_addrs=127.0.115.190:44475
--webserver_port=35687
--webserver_interface=127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab
--principal=oryx/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.189@KRBTEST.COM
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.188@KRBTEST.COM
I20260924 20:08:23.331987 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.315006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:56753 (local address 127.0.115.190:44475)
0924 20:08:23.315137 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.315142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.316208 (+ 1066us) server_negotiation.cc:407] Connection header received
0924 20:08:23.317457 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.317465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.317533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.317592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.320307 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.321223 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.322711 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.323139 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.323233 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.327703 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.327708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.327710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.327739 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.329680 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330251 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.330256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.330257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.330344 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330651 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.330655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.330658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.330914 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.330994 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.331656 (+ 662us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.331796 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:08:23.332182 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.315001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:45199 (local address 127.0.115.189:38309)
0924 20:08:23.315133 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.315138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.316454 (+ 1316us) server_negotiation.cc:407] Connection header received
0924 20:08:23.317865 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.317871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.317935 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.318009 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.320307 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.320852 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.322692 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.323085 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.323164 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.327311 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.327318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.327324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.327361 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.329295 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330005 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.330012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.330017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.330102 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330522 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.330528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.330533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.330806 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.330894 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.331682 (+ 788us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.331814 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:08:23.332396 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.316921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:41189 (local address 127.0.115.188:34931)
0924 20:08:23.318412 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.318417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.318431 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:23.318477 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.318481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.318532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.319221 (+ 689us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.321205 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.322266 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.323435 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.323773 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.325544 (+ 1771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.327700 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.327704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.327707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.327735 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.329834 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330456 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.330459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.330462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.330534 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330854 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.330857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.330859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.331124 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.331207 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.332078 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.332210 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
I20260924 20:08:23.333581 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.315439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:56753)
0924 20:08:23.316043 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.316093 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.317190 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.318482 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.318488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.318983 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.320095 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.320117 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.321381 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.321388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.322561 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.322575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.323423 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.323429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.323584 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.324367 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.324404 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.327536 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.329828 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.329831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.329833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.330131 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.330452 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.330457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.330538 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.331052 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.331055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.331347 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.332211 (+ 864us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.333010 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":961,"spinlock_wait_cycles":55808,"thread_start_us":169,"threads_started":1}
I20260924 20:08:23.334039 7560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.316537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.115.129:41189)
0924 20:08:23.316781 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.316817 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.317196 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.318683 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.318686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.318981 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.321055 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.321063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.322406 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.322412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.323298 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.323308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.324391 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.324395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.324493 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.324966 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.324981 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.327536 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.330057 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.330067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.330321 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.330671 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.330675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.330747 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.331353 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.331356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.331428 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.331580 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.332593 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1542,"spinlock_wait_cycles":13184,"thread_start_us":82,"threads_started":1}
I20260924 20:08:23.334506 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.315439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.115.129:45199)
0924 20:08:23.316365 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.316380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.317196 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.318317 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.318329 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.318981 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.320095 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.320117 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.321047 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.321050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.322547 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.322557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.323388 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.323393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.323581 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.324394 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.324409 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.326991 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.329433 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.329442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.329463 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.329882 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.330214 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.330220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.330225 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.330412 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.330966 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.330975 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.331346 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.332410 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.332741 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":531,"spinlock_wait_cycles":44160,"thread_start_us":699,"threads_started":1}
I20260924 20:08:23.335024 7554 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44475
I20260924 20:08:23.335029 7555 heartbeater.cc:347] Connected to a master server at 127.0.115.188:34931
I20260924 20:08:23.335304 7554 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.335332 7556 heartbeater.cc:347] Connected to a master server at 127.0.115.189:38309
I20260924 20:08:23.335417 7556 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.335304 7555 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.336041 7554 heartbeater.cc:511] Master 127.0.115.190:44475 requested a full tablet report, sending...
I20260924 20:08:23.336038 7555 heartbeater.cc:511] Master 127.0.115.188:34931 requested a full tablet report, sending...
I20260924 20:08:23.336038 7556 heartbeater.cc:511] Master 127.0.115.189:38309 requested a full tablet report, sending...
I20260924 20:08:23.337572 6585 ts_manager.cc:194] Re-registered known tserver with Master: 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
I20260924 20:08:23.337731 7192 ts_manager.cc:194] Registered new tserver with Master: 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
I20260924 20:08:23.338337 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:56753
I20260924 20:08:23.338713 7375 ts_manager.cc:194] Registered new tserver with Master: 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
W20260924 20:08:23.445914 7562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:23.446228 7562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:23.446300 7562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:23.450529 7562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:23.450627 7562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:23.450726 7562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:23.456389 7562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab
--principal=oryx/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:44049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=35687
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:23.457453 7562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:23.458479 7562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:23.468613 7568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:23.468806 7562 server_base.cc:1034] running on GCE node
W20260924 20:08:23.468613 7570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:23.468863 7567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:23.469307 7562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:23.469941 7562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:23.471128 7562 hybrid_clock.cc:648] HybridClock initialized: now 1790280503471124 us; error 41 us; skew 500 ppm
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:23.474231 7562 init.cc:377] Logged in from keytab as oryx/127.0.115.130@KRBTEST.COM (short username oryx)
I20260924 20:08:23.475116 7562 webserver.cc:460] Webserver started at http://127.0.115.130:35687/ using document root <none> and password file <none>
I20260924 20:08:23.475374 7044 heartbeater.cc:503] Master 127.0.115.190:44475 was elected leader, sending a full tablet report...
I20260924 20:08:23.475730 7562 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:23.475847 7562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:23.480090 7562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:08:23.482309 7577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.484231 7562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:23.484387 7562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "8fc75e3aa684468382372fa979d43f85"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "df8783f5c5b09fad48414a995370fc66"
server_key_iv: "24804b439207f7d60982490058204832"
server_key_version: "encryptionkey@0"
I20260924 20:08:23.485258 7562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:23.507011 7562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:23.510226 7562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:23.510439 7562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:23.511076 7562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:23.512003 7562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:23.512076 7562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.512151 7562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:23.512197 7562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.521924 7562 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:44049
I20260924 20:08:23.521963 7690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:44049 every 8 connection(s)
I20260924 20:08:23.522989 7562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:23.524853 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 7562
I20260924 20:08:23.525038 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6915
I20260924 20:08:23.534829 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:43971
--local_ip_for_outbound_sockets=127.0.115.131
--tserver_master_addrs=127.0.115.190:44475
--webserver_port=44499
--webserver_interface=127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab
--principal=oryx/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.188@KRBTEST.COM
I20260924 20:08:23.536346 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.527542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.0.1:39996)
0924 20:08:23.527867 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.527881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.527977 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.529678 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.529684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.529707 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:23.529827 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.529837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.532573 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.532580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.533910 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.533925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.535774 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.535780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.535908 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.535970 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.536028 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.536148 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.189@KRBTEST.COM
I20260924 20:08:23.537180 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.528202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.0.115.188:34931)
0924 20:08:23.529354 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.529360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.529377 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:08:23.529430 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.529435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.529497 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.529936 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:23.531278 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.532446 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.534066 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.535340 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.536671 (+ 1331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.536855 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.536896 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.537006 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":81,"threads_started":1}
I20260924 20:08:23.537776 7384 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:23.539709 7414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3766a2ab094b89b7abed98b7412ac0. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.539886 7414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: This master's current role is: LEARNER
I20260924 20:08:23.542995 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.526711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:60523 (local address 127.0.115.189:38309)
0924 20:08:23.527015 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.527020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.527037 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:08:23.527596 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.527599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.527658 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.527717 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.530106 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.530951 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.533014 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.533356 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.533434 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.538326 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.538330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.538333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.538361 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.540174 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.541078 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.541084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.541087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.541173 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.541623 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.541630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.541634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.541866 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.541957 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.542708 (+ 751us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.542837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260924 20:08:23.542995 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.527671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:45703 (local address 127.0.115.190:44475)
0924 20:08:23.528114 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.528117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.528567 (+ 450us) server_negotiation.cc:407] Connection header received
0924 20:08:23.528746 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.528749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.528801 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.528857 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.530231 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.530993 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.534543 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.534878 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.534969 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.538698 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.538702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.538705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.538763 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.540818 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.541313 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.541318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.541321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.541410 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.541704 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.541707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.541709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.541916 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.541991 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.542708 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.542837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260924 20:08:23.543814 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.525669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:48091 (local address 127.0.115.188:34931)
0924 20:08:23.525868 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.525875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.527678 (+ 1803us) server_negotiation.cc:407] Connection header received
0924 20:08:23.527869 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.527876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.527961 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.528022 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.530058 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.530956 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.533014 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.533370 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.533434 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.537771 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.537775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.537778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.537805 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.541514 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.542120 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.542124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.542125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.542212 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.542603 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.542607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.542609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.542866 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.542952 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.543450 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.543600 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:08:23.544843 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.525396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.115.130:48091)
0924 20:08:23.527584 (+ 2188us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.527601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.527755 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.528204 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.528216 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.528875 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.529932 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.529952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.531137 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.531143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.532874 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.532886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.533663 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.533667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.533887 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.534508 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.534532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.537549 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.541656 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.541661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.541665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.541982 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.542325 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.542330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.542335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.542469 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.543038 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.543044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.543255 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.544030 (+ 775us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.544217 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2074,"mutex_wait_us":844,"spinlock_wait_cycles":111616,"thread_start_us":137,"threads_started":1}
I20260924 20:08:23.544840 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.526798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:45703)
0924 20:08:23.528482 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.528497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.528634 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.529002 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.529006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.529177 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.530111 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.530118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.531144 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.531148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.534393 (+ 3245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.534407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.535132 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.535135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.535241 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.537146 (+ 1905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.537161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.538541 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.540945 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.540947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.540949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.541178 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.541517 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.541520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.541522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.541606 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.542040 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.542042 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.542411 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.543158 (+ 747us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.543646 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"mutex_wait_us":444,"spinlock_wait_cycles":13312,"thread_start_us":80,"threads_started":1}
I20260924 20:08:23.545107 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.525395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.115.130:60523)
0924 20:08:23.526298 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.526353 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.527349 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.528203 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.528212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.528870 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.529932 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.529952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.531144 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.531148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.532874 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.532886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.533663 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.533667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.534245 (+ 578us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.535660 (+ 1415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.535681 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.538163 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.540576 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.540586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.540604 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.540964 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.541335 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.541338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.541340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.541517 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.542032 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.542038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.542411 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.543191 (+ 780us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.543696 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":734,"spinlock_wait_cycles":78080,"thread_start_us":714,"threads_started":1}
I20260924 20:08:23.546211 7693 heartbeater.cc:347] Connected to a master server at 127.0.115.189:38309
I20260924 20:08:23.546281 7692 heartbeater.cc:347] Connected to a master server at 127.0.115.188:34931
I20260924 20:08:23.546379 7691 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44475
I20260924 20:08:23.546463 7693 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.546463 7692 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.546482 7691 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.547113 7691 heartbeater.cc:511] Master 127.0.115.190:44475 requested a full tablet report, sending...
I20260924 20:08:23.547103 7693 heartbeater.cc:511] Master 127.0.115.189:38309 requested a full tablet report, sending...
I20260924 20:08:23.547103 7692 heartbeater.cc:511] Master 127.0.115.188:34931 requested a full tablet report, sending...
I20260924 20:08:23.548412 7375 ts_manager.cc:194] Registered new tserver with Master: 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049)
I20260924 20:08:23.548468 7192 ts_manager.cc:194] Registered new tserver with Master: 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049)
I20260924 20:08:23.549068 6585 ts_manager.cc:194] Re-registered known tserver with Master: 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049)
I20260924 20:08:23.549645 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:45703
W20260924 20:08:23.655035 7699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:23.655304 7699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:23.655346 7699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:23.659272 7699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:23.659328 7699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:23.659384 7699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:23.664952 7699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab
--principal=oryx/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:43971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=44499
--tserver_master_addrs=127.0.115.190:44475,127.0.115.189:38309,127.0.115.188:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:23.665967 7699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:23.666980 7699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:23.677335 7707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:23.677322 7704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:23.677345 7705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:23.677528 7699 server_base.cc:1034] running on GCE node
I20260924 20:08:23.677994 7699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:23.678726 7699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:23.679898 7699 hybrid_clock.cc:648] HybridClock initialized: now 1790280503679862 us; error 45 us; skew 500 ppm
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:23.682871 7699 init.cc:377] Logged in from keytab as oryx/127.0.115.131@KRBTEST.COM (short username oryx)
I20260924 20:08:23.683799 7699 webserver.cc:460] Webserver started at http://127.0.115.131:44499/ using document root <none> and password file <none>
I20260924 20:08:23.684424 7699 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:23.684520 7699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:23.688524 7699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:08:23.690646 7714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.692008 7699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:23.692121 7699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "720ffa94430342fb91f1b1c247992fce"
format_stamp: "Formatted at 2026-09-24 20:08:22 on dist-test-slave-2pm7"
server_key: "67f20b74179de4c04e0eead2cd82ad97"
server_key_iv: "ce009f56e0e979038ce98fa5c7ee1c3f"
server_key_version: "encryptionkey@0"
I20260924 20:08:23.692659 7699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:23.714104 7699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:23.717708 7699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:23.717917 7699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:23.718611 7699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:23.719507 7699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:23.719559 7699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.719635 7699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:23.719664 7699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:23.729199 7699 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:43971
I20260924 20:08:23.729283 7827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:43971 every 8 connection(s)
I20260924 20:08:23.730402 7699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:23.733390 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 7699
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.188@KRBTEST.COM
Sep 24 20:08:23 dist-test-slave-2pm7 krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280503, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.189@KRBTEST.COM
I20260924 20:08:23.747443 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.734594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:54663 (local address 127.0.115.190:44475)
0924 20:08:23.734723 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.734727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.735486 (+ 759us) server_negotiation.cc:407] Connection header received
0924 20:08:23.735674 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.735677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.735724 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.735776 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.737719 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.738487 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.739750 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.740128 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.740195 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.743381 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.743385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.743388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.743417 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.745205 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.745778 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.745782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.745784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.745865 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.746240 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.746243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.746245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.746451 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.746515 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.747166 (+ 651us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.747286 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:08:23.748533 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.732533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:52877 (local address 127.0.115.189:38309)
0924 20:08:23.732941 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.732947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.734888 (+ 1941us) server_negotiation.cc:407] Connection header received
0924 20:08:23.735342 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.735345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.735402 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.735484 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.737075 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.737908 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.739768 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.740140 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.740241 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.744418 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.744422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.744425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.744451 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.746515 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.747136 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.747139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.747142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.747219 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.747537 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.747540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.747542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.747747 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.747824 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.748274 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.748387 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260924 20:08:23.749732 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.733028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.115.131:39417)
0924 20:08:23.733854 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.733909 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.735006 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.735700 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.735709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.737223 (+ 1514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.737336 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.737342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.738351 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.738354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.739619 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.739631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.740392 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.740396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.740988 (+ 592us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.741622 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.741641 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.744260 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.746625 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.746628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.746631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.747011 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.747324 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.747327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.747329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.747429 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.747887 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.747890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.748060 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.748210 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.748598 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":527,"spinlock_wait_cycles":23552,"thread_start_us":770,"threads_started":1}
I20260924 20:08:23.749982 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.733028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.115.131:52877)
0924 20:08:23.734777 (+ 1749us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.734792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.735011 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.735704 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.735710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.736255 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.736916 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.736933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.738077 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.738080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.739650 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.739659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.740392 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.740396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.740604 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.741402 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.741424 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.744260 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.746625 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.746628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.746631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.747011 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.747330 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.747333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.747335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.747435 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.747891 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.747893 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.748069 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.748729 (+ 660us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.748856 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394,"mutex_wait_us":215,"spinlock_wait_cycles":19968,"thread_start_us":163,"threads_started":1}
I20260924 20:08:23.750622 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.732786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:39417 (local address 127.0.115.188:34931)
0924 20:08:23.732934 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.732939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.734013 (+ 1074us) server_negotiation.cc:407] Connection header received
0924 20:08:23.735230 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.735233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.735292 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.735375 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.737461 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.738224 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.739768 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.740154 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.740241 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.744418 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.744422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.744425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.744451 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.746477 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.747130 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.747133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.747135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.747219 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.747537 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.747540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.747542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.747756 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.747839 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.750318 (+ 2479us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.750442 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:08:23.750643 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.734229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:54663)
0924 20:08:23.735407 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.735420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.735563 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.735923 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.735926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.737511 (+ 1585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:23.737601 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.737607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.738627 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.738630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.739619 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.739628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.740390 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.740393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.740607 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.741328 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:23.741350 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:23.743168 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:23.745330 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.745336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.745353 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.745665 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.745962 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:23.745965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:23.745967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:23.746130 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:23.746592 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:23.746598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:23.746892 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.750322 (+ 3430us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.750471 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":46,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260924 20:08:23.751132 7828 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44475
I20260924 20:08:23.751389 7828 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.751878 7828 heartbeater.cc:511] Master 127.0.115.190:44475 requested a full tablet report, sending...
I20260924 20:08:23.751108 7830 heartbeater.cc:347] Connected to a master server at 127.0.115.189:38309
I20260924 20:08:23.752466 7830 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.752669 7830 heartbeater.cc:511] Master 127.0.115.189:38309 requested a full tablet report, sending...
I20260924 20:08:23.753012 7829 heartbeater.cc:347] Connected to a master server at 127.0.115.188:34931
I20260924 20:08:23.753026 6585 ts_manager.cc:194] Re-registered known tserver with Master: 720ffa94430342fb91f1b1c247992fce (127.0.115.131:43971)
I20260924 20:08:23.753087 7829 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:23.753229 7829 heartbeater.cc:511] Master 127.0.115.188:34931 requested a full tablet report, sending...
I20260924 20:08:23.753930 7375 ts_manager.cc:194] Registered new tserver with Master: 720ffa94430342fb91f1b1c247992fce (127.0.115.131:43971)
I20260924 20:08:23.754204 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:54663
I20260924 20:08:23.754362 7192 ts_manager.cc:194] Registered new tserver with Master: 720ffa94430342fb91f1b1c247992fce (127.0.115.131:43971)
I20260924 20:08:23.758500 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.735931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.0.115.190:44475)
0924 20:08:23.736292 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.736296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.749010 (+ 12714us) server_negotiation.cc:407] Connection header received
0924 20:08:23.749205 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.749208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.749264 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.749323 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.750317 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.751083 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.752250 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.752681 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.752791 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.754816 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.754822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.754826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.754856 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.756659 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.757145 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.757149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.757151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.757237 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.757529 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.757536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.757538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.757789 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.757949 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.758182 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.758311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":68,"threads_started":1}
I20260924 20:08:23.763710 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.748572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.0.115.188:34931)
0924 20:08:23.748719 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.748723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.752751 (+ 4028us) server_negotiation.cc:407] Connection header received
0924 20:08:23.752927 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.752929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.752972 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.753024 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.755276 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.756152 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.757656 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.758001 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.758138 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.759760 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.759763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.759765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.759787 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.761842 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.762381 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.762385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.762387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.762471 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.762778 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.762781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.762783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.762981 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.763079 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.763426 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.763556 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:08:23.763710 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.736343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.0.115.189:38309)
0924 20:08:23.736562 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.736576 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.752532 (+ 15956us) server_negotiation.cc:407] Connection header received
0924 20:08:23.754219 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.754223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.754287 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.754357 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:23.755228 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.756002 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.756966 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.757215 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.757332 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.759713 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:23.759717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:23.759721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:23.759844 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:23.761960 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.762414 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.762418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.762419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.762509 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:23.762792 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:23.762796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:23.762798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:23.762980 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:23.763154 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.763391 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.763514 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260924 20:08:23.762554 6585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:23.766858 6585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:23.770253 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: NON_VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: true } } }
I20260924 20:08:23.770403 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:23.773952 7847 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: attempting to promote NON_VOTER 0f7bc2a5c89b45b5854effb879ee1512 to VOTER
I20260924 20:08:23.774274 7847 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:23.775380 7384 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 LEARNER]: Refusing update from remote peer ab3766a2ab094b89b7abed98b7412ac0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 20:08:23.775453 7202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Refusing update from remote peer ab3766a2ab094b89b7abed98b7412ac0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 20:08:23.775872 7846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 20:08:23.776134 7847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 20:08:23.777429 7846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.777889 7202 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.778941 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.779052 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:23.778963 7846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.779354 7846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:23.779667 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.779796 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:08:23.780939 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.781049 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:23.781675 7384 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.783661 7414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab3766a2ab094b89b7abed98b7412ac0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:23.783790 7414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:08:23.789906 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.783278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:44821 (local address 127.0.0.1:37830)
0924 20:08:23.784825 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.784859 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.784964 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.785587 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.785590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.785611 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:23.786510 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.786521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.787515 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.787520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.788383 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.788392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.789471 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.789473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.789589 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.789614 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.789686 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.789760 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"thread_start_us":116,"threads_started":1}
I20260924 20:08:23.790390 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.783734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.0.115.129:44821)
0924 20:08:23.784680 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.784696 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.784970 (+ 274us) server_negotiation.cc:407] Connection header received
0924 20:08:23.785099 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.785105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.785346 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.785602 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:23.786674 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.787396 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.788502 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.789134 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.789241 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.789419 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.790191 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.790287 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":88,"threads_started":1}
I20260924 20:08:23.790508 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.782971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:44049 (local address 127.0.0.1:59534)
0924 20:08:23.783136 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.783153 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.783236 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.786188 (+ 2952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.786192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.786211 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:23.786304 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.786312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.787767 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.787770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.788982 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.788991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.790144 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.790148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.790232 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.790262 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.790312 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.790386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":18}
I20260924 20:08:23.791265 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.783177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.0.115.130:44049)
0924 20:08:23.784023 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.784040 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.784067 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:08:23.784154 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.784160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.786051 (+ 1891us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.786213 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:23.786443 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.787638 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.789148 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.789996 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.790822 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.790965 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.791017 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.791110 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":135,"threads_started":1}
I20260924 20:08:23.791862 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.784573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:43971 (local address 127.0.0.1:53884)
0924 20:08:23.786459 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.786472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.786574 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.787235 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.787238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.787256 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:23.787335 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.787340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.788745 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.788748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.790260 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.790267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.791464 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.791473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.791539 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.791564 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.791663 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.791733 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"mutex_wait_us":26,"thread_start_us":105,"threads_started":1}
I20260924 20:08:23.793499 7488 tablet_service.cc:1467] Processing CreateTablet for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c (DEFAULT_TABLE table=test-table [id=d099ec3bacfd43749a8502738c4fc524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:23.793916 7625 tablet_service.cc:1467] Processing CreateTablet for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c (DEFAULT_TABLE table=test-table [id=d099ec3bacfd43749a8502738c4fc524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:23.794427 7488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:23.794698 7625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:23.796048 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.785096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.115.131:43971)
0924 20:08:23.785411 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.785423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.786671 (+ 1248us) server_negotiation.cc:407] Connection header received
0924 20:08:23.786753 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.786758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.786945 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.787098 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:23.787450 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.788604 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.790384 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.791155 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.791255 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.791399 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.791697 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.791854 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260924 20:08:23.798921 7762 tablet_service.cc:1467] Processing CreateTablet for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c (DEFAULT_TABLE table=test-table [id=d099ec3bacfd43749a8502738c4fc524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:23.800047 7762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:23.804359 7862 tablet_bootstrap.cc:492] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Bootstrap starting.
I20260924 20:08:23.806360 7862 tablet_bootstrap.cc:654] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:23.807152 7862 log.cc:824] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:23.807824 7863 tablet_bootstrap.cc:492] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Bootstrap starting.
I20260924 20:08:23.807945 7861 tablet_bootstrap.cc:492] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Bootstrap starting.
I20260924 20:08:23.809734 7861 tablet_bootstrap.cc:654] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:23.809741 7863 tablet_bootstrap.cc:654] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:23.810494 7861 log.cc:824] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:23.810566 7863 log.cc:824] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:23.811800 7862 tablet_bootstrap.cc:492] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: No bootstrap required, opened a new log
I20260924 20:08:23.812063 7862 ts_tablet_manager.cc:1397] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 20:08:23.812585 7861 tablet_bootstrap.cc:492] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: No bootstrap required, opened a new log
I20260924 20:08:23.812587 7863 tablet_bootstrap.cc:492] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: No bootstrap required, opened a new log
I20260924 20:08:23.812780 7863 ts_tablet_manager.cc:1397] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:08:23.812780 7861 ts_tablet_manager.cc:1397] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:08:23.816162 7862 raft_consensus.cc:348] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.816416 7862 raft_consensus.cc:374] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:23.816469 7862 raft_consensus.cc:729] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc75e3aa684468382372fa979d43f85, State: Initialized, Role: FOLLOWER
I20260924 20:08:23.816929 7862 consensus_queue.cc:260] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.817615 7691 heartbeater.cc:503] Master 127.0.115.190:44475 was elected leader, sending a full tablet report...
I20260924 20:08:23.817731 7863 raft_consensus.cc:348] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.817847 7861 raft_consensus.cc:348] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.817993 7862 ts_tablet_manager.cc:1428] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:08:23.817996 7863 raft_consensus.cc:374] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:23.818046 7863 raft_consensus.cc:729] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720ffa94430342fb91f1b1c247992fce, State: Initialized, Role: FOLLOWER
I20260924 20:08:23.818068 7861 raft_consensus.cc:374] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:23.818114 7861 raft_consensus.cc:729] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a0ac3b29d3410baa2befb435002333, State: Initialized, Role: FOLLOWER
I20260924 20:08:23.818521 7863 consensus_queue.cc:260] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.818519 7861 consensus_queue.cc:260] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.819496 7861 ts_tablet_manager.cc:1428] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:08:23.819496 7863 ts_tablet_manager.cc:1428] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:08:23.820168 7828 heartbeater.cc:503] Master 127.0.115.190:44475 was elected leader, sending a full tablet report...
I20260924 20:08:23.821810 7554 heartbeater.cc:503] Master 127.0.115.190:44475 was elected leader, sending a full tablet report...
I20260924 20:08:23.965550 7869 raft_consensus.cc:482] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:23.965781 7869 raft_consensus.cc:504] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.967010 7869 leader_election.cc:288] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049), 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
I20260924 20:08:23.972303 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.967516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:58671 (local address 127.0.115.129:44821)
0924 20:08:23.967676 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.967680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.967763 (+ 83us) server_negotiation.cc:407] Connection header received
0924 20:08:23.967858 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.967861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.967911 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.968014 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:23.968268 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.969130 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.970900 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.971500 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.971629 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.971685 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.972063 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.972167 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:08:23.972302 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.967366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:34517 (local address 127.0.115.130:44049)
0924 20:08:23.967519 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:23.967525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:23.967566 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:08:23.967662 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:23.967665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:23.967731 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:23.967824 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:23.968185 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.969131 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.970900 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.971500 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.971637 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.971685 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:23.972066 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 20:08:23.972166 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:08:23.972313 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.967215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:44049 (local address 127.0.115.131:34517)
0924 20:08:23.967376 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.967393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.967490 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.967865 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.967868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.967881 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:23.968061 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.968069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.969267 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.969271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.970742 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.970754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.971636 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.971640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.971808 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.971930 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.972028 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.972125 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":175}
I20260924 20:08:23.972311 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:23.967373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:44821 (local address 127.0.115.131:58671)
0924 20:08:23.967671 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:23.967686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:23.967760 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:23.968040 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:23.968043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:23.968056 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:23.968158 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.968164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.969267 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.969271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.970742 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:23.970754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:23.971669 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:23.971673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:23.971802 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:23.971930 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 20:08:23.972030 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 20:08:23.972125 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":203}
I20260924 20:08:23.973384 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b9c7ac1548481bb7cfe9ffaec27b4c" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc75e3aa684468382372fa979d43f85" is_pre_election: true
I20260924 20:08:23.973575 7508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b9c7ac1548481bb7cfe9ffaec27b4c" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a0ac3b29d3410baa2befb435002333" is_pre_election: true
I20260924 20:08:23.973831 7645 raft_consensus.cc:2454] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 0.
I20260924 20:08:23.973867 7508 raft_consensus.cc:2454] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 0.
I20260924 20:08:23.974390 7716 leader_election.cc:302] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a0ac3b29d3410baa2befb435002333, 720ffa94430342fb91f1b1c247992fce; no voters:
I20260924 20:08:23.974684 7869 raft_consensus.cc:2781] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:23.974777 7869 raft_consensus.cc:482] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:23.974821 7869 raft_consensus.cc:3037] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:23.975898 7869 raft_consensus.cc:504] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:23.976395 7869 leader_election.cc:288] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 election: Requested vote from peers 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049), 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
I20260924 20:08:23.976747 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b9c7ac1548481bb7cfe9ffaec27b4c" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc75e3aa684468382372fa979d43f85"
I20260924 20:08:23.976804 7508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b9c7ac1548481bb7cfe9ffaec27b4c" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a0ac3b29d3410baa2befb435002333"
I20260924 20:08:23.976907 7508 raft_consensus.cc:3037] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:23.976894 7645 raft_consensus.cc:3037] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:23.977898 7645 raft_consensus.cc:2454] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 1.
I20260924 20:08:23.977980 7508 raft_consensus.cc:2454] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 1.
I20260924 20:08:23.978302 7716 leader_election.cc:302] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a0ac3b29d3410baa2befb435002333, 720ffa94430342fb91f1b1c247992fce; no voters:
I20260924 20:08:23.978508 7869 raft_consensus.cc:2781] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:23.978745 7869 raft_consensus.cc:686] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 1 LEADER]: Becoming Leader. State: Replica: 720ffa94430342fb91f1b1c247992fce, State: Running, Role: LEADER
I20260924 20:08:23.979113 7869 consensus_queue.cc:237] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
W20260924 20:08:23.981947 7831 tablet.cc:2369] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:23.983597 6584 catalog_manager.cc:5495] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 720ffa94430342fb91f1b1c247992fce (127.0.115.131). New cstate: current_term: 1 leader_uuid: "720ffa94430342fb91f1b1c247992fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } health_report { overall_health: HEALTHY } } }
W20260924 20:08:24.024808 7694 tablet.cc:2369] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:24.027899 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:24.024318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.0.115.131:43971)
0924 20:08:24.024463 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:24.024467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:24.024480 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:24.024558 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:24.024560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:24.024635 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:24.024777 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:24.025028 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.025564 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.026442 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.026666 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.026787 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:24.027083 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:24.027507 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:24.027648 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:24.027689 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 20:08:24.027747 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:08:24.034312 7645 raft_consensus.cc:1261] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Refusing update from remote peer 720ffa94430342fb91f1b1c247992fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:24.034324 7508 raft_consensus.cc:1261] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Refusing update from remote peer 720ffa94430342fb91f1b1c247992fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:24.035099 7870 consensus_queue.cc:1035] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:24.035291 7869 consensus_queue.cc:1035] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:24.047672 7874 mvcc.cc:205] Tried to move back new op lower bound from 7332988944516603904 to 7332988944301494272. Current Snapshot: MvccSnapshot[applied={T|T < 7332988944516603904}]
I20260924 20:08:24.049206 7875 mvcc.cc:205] Tried to move back new op lower bound from 7332988944516603904 to 7332988944301494272. Current Snapshot: MvccSnapshot[applied={T|T < 7332988944516603904}]
I20260924 20:08:24.054833 6584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:24.055024 6584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:24.059674 6584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab3766a2ab094b89b7abed98b7412ac0: Sending DeleteTablet for 3 replicas of tablet 17b9c7ac1548481bb7cfe9ffaec27b4c
I20260924 20:08:24.060930 7488 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:24 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:37830
I20260924 20:08:24.060948 7625 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:24 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:59534
I20260924 20:08:24.061121 7762 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b9c7ac1548481bb7cfe9ffaec27b4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:24 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:53884
I20260924 20:08:24.061481 7883 tablet_replica.cc:330] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: stopping tablet replica
I20260924 20:08:24.061564 7885 tablet_replica.cc:330] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: stopping tablet replica
I20260924 20:08:24.061939 7883 raft_consensus.cc:2229] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:24.062170 7885 raft_consensus.cc:2229] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:24.062521 7883 raft_consensus.cc:2258] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:24.062553 7885 raft_consensus.cc:2258] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:24.064524 7885 ts_tablet_manager.cc:1905] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:24.064524 7883 ts_tablet_manager.cc:1905] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:24.064947 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 6553
I20260924 20:08:24.066273 7884 tablet_replica.cc:330] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: stopping tablet replica
I20260924 20:08:24.066625 7884 raft_consensus.cc:2229] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:24.066942 7884 raft_consensus.cc:2258] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:24.068562 7884 ts_tablet_manager.cc:1905] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:24.068728 7883 ts_tablet_manager.cc:1918] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:24.068828 7883 log.cc:1196] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/17b9c7ac1548481bb7cfe9ffaec27b4c
I20260924 20:08:24.069147 7883 ts_tablet_manager.cc:1939] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 8fc75e3aa684468382372fa979d43f85: Deleting consensus metadata
I20260924 20:08:24.072073 7884 ts_tablet_manager.cc:1918] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:24.072177 7884 log.cc:1196] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/17b9c7ac1548481bb7cfe9ffaec27b4c
I20260924 20:08:24.072629 7884 ts_tablet_manager.cc:1939] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 69a0ac3b29d3410baa2befb435002333: Deleting consensus metadata
W20260924 20:08:24.074270 7442 connection.cc:535] server connection from 127.0.0.1:37830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 20:08:24.074697 7579 connection.cc:535] server connection from 127.0.0.1:59534 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:08:24.074817 7885 ts_tablet_manager.cc:1918] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:24.074910 7885 log.cc:1196] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/17b9c7ac1548481bb7cfe9ffaec27b4c
I20260924 20:08:24.075306 7885 ts_tablet_manager.cc:1939] T 17b9c7ac1548481bb7cfe9ffaec27b4c P 720ffa94430342fb91f1b1c247992fce: Deleting consensus metadata
W20260924 20:08:24.076442 7718 connection.cc:441] server connection from 127.0.0.1:53884 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53884 (request call id 1) could send its response
I20260924 20:08:24.083352 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:24.063786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.0.115.189:38309)
0924 20:08:24.063917 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:24.063924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:24.063962 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:08:24.064010 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:24.064014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:24.064067 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:24.064141 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:24.073471 (+ 9330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.074333 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.076114 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.076468 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.076559 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:24.079118 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:24.079124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:24.079130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:24.079168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:24.081119 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.081639 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.081643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.081645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.081745 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.082036 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.082040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.082042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.082278 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:24.082383 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:24.083029 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 20:08:24.083172 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:08:24.088671 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:24.077920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.0.115.189:38309)
0924 20:08:24.078056 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:24.078060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:24.078744 (+ 684us) server_negotiation.cc:407] Connection header received
0924 20:08:24.078929 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:24.078932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:24.078978 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:24.079027 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:24.079852 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.080665 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.081879 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.082228 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.082310 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:24.084861 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:24.084865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:24.084868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:24.084892 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:24.086800 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.087299 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.087303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.087305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.087385 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.087675 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.087678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.087680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.087891 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:24.088014 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:24.088379 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 20:08:24.088499 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:08:24.089267 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:24.078843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.0.115.188:34931)
0924 20:08:24.078966 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:24.078970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:24.079821 (+ 851us) server_negotiation.cc:407] Connection header received
0924 20:08:24.080055 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:24.080060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:24.080149 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:24.080207 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:24.081264 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.082035 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.083250 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.083561 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.083643 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:24.085484 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:24.085488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:24.085491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:24.085518 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:24.087254 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.087721 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.087725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.087726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.087809 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.088185 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.088190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.088193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.088396 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:24.088507 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:24.088904 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 20:08:24.089039 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:08:24.093087 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:24.062923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.0.115.188:34931)
0924 20:08:24.063035 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:24.063042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:24.083989 (+ 20947us) server_negotiation.cc:407] Connection header received
0924 20:08:24.084493 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:24.084497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:24.084557 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:24.084629 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:24.085464 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.086275 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.087553 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:24.087922 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:24.088067 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:24.089701 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:24.089706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:24.089713 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:24.089745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:24.091427 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.091849 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.091851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.091852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.091921 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:24.092258 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:24.092262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:24.092264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:24.092450 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:24.092587 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:24.092806 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 20:08:24.092926 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 20:08:25.049916 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.049416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:42499)
0924 20:08:25.049707 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.049808 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260924 20:08:25.050067 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.049563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:59829)
0924 20:08:25.049871 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.049951 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":150,"threads_started":1}
W20260924 20:08:25.050284 7554 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:44475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
W20260924 20:08:25.050379 7691 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:44475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:25.054080 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.053702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:56059)
0924 20:08:25.053924 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.053996 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
W20260924 20:08:25.054364 7828 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:44475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:25.560777 7897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab3766a2ab094b89b7abed98b7412ac0)
I20260924 20:08:25.561015 7897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:25.561327 7898 raft_consensus.cc:482] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab3766a2ab094b89b7abed98b7412ac0)
I20260924 20:08:25.561561 7898 raft_consensus.cc:504] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:25.562244 7897 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475), da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189:38309)
I20260924 20:08:25.562534 7898 leader_election.cc:288] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475), 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188:34931)
I20260924 20:08:25.562899 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.562380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57036)
0924 20:08:25.562707 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.562789 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":89,"threads_started":1}
W20260924 20:08:25.563084 7362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:25.563205 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.562708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57048)
0924 20:08:25.562983 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.563069 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260924 20:08:25.564033 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.563707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57052)
0924 20:08:25.563851 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.563921 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 20:08:25.564034 7179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
W20260924 20:08:25.564165 7362 leader_election.cc:334] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475): Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:25.565231 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.564857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57066)
0924 20:08:25.565104 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.565154 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
W20260924 20:08:25.565335 7179 leader_election.cc:334] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475): Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:25.567427 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.562515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:38309 (local address 127.0.0.1:44182)
0924 20:08:25.562865 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.562882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:25.563004 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:25.564305 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:25.564308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:25.564326 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:25.564452 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:25.564459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.565434 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.565438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:25.566235 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:25.566244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.566864 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.566868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:25.567034 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:25.567064 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:08:25.567168 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 20:08:25.567245 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":57,"spinlock_wait_cycles":2432,"thread_start_us":88,"threads_started":1}
I20260924 20:08:25.567443 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.563807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.0.115.189:38309)
0924 20:08:25.564048 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:25.564054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:25.564070 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:25.564118 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:25.564122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:25.564165 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:25.564565 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:25.564655 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.565304 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.566369 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.566749 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.566874 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:25.566933 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:25.567190 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 20:08:25.567274 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260924 20:08:25.567574 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.563392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:34931 (local address 127.0.0.1:40048)
0924 20:08:25.563497 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.563508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:25.563611 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:25.564051 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:25.564058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:25.564084 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:25.564221 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:25.564232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.565414 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.565418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:25.566369 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:25.566379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.567199 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.567203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:25.567287 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:25.567307 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:08:25.567385 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 20:08:25.567449 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 20:08:25.567658 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.563253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.0.115.188:34931)
0924 20:08:25.563525 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:25.563536 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:25.563554 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:08:25.563802 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:25.563807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:25.563859 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:25.564351 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:25.564476 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.565249 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.566510 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:25.567085 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:25.567189 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:25.567244 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:25.567417 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 20:08:25.567501 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":211,"spinlock_wait_cycles":3712,"thread_start_us":111,"threads_started":1}
I20260924 20:08:25.568006 7202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f7bc2a5c89b45b5854effb879ee1512" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da565a0436144a6895b0bf8e0dbb1603" is_pre_election: true
I20260924 20:08:25.568236 7202 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f7bc2a5c89b45b5854effb879ee1512 in term 1.
I20260924 20:08:25.568212 7384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da565a0436144a6895b0bf8e0dbb1603" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0f7bc2a5c89b45b5854effb879ee1512" is_pre_election: true
I20260924 20:08:25.568444 7384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da565a0436144a6895b0bf8e0dbb1603 in term 1.
I20260924 20:08:25.568609 7359 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f7bc2a5c89b45b5854effb879ee1512, da565a0436144a6895b0bf8e0dbb1603; no voters: ab3766a2ab094b89b7abed98b7412ac0
I20260924 20:08:25.568833 7897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 20:08:25.568890 7897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab3766a2ab094b89b7abed98b7412ac0)
I20260924 20:08:25.568826 7179 leader_election.cc:302] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f7bc2a5c89b45b5854effb879ee1512, da565a0436144a6895b0bf8e0dbb1603; no voters: ab3766a2ab094b89b7abed98b7412ac0
I20260924 20:08:25.568930 7897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:08:25.569067 7898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 20:08:25.569131 7898 raft_consensus.cc:482] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab3766a2ab094b89b7abed98b7412ac0)
I20260924 20:08:25.569177 7898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:08:25.570209 7897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:25.570335 7898 raft_consensus.cc:504] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:25.570609 7897 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 2 election: Requested vote from peers ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475), da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189:38309)
I20260924 20:08:25.570650 7898 leader_election.cc:288] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [CANDIDATE]: Term 2 election: Requested vote from peers ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475), 0f7bc2a5c89b45b5854effb879ee1512 (127.0.115.188:34931)
I20260924 20:08:25.571008 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.570758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57076)
0924 20:08:25.570879 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.570932 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 20:08:25.571163 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.570938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57082)
0924 20:08:25.571058 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.571107 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 20:08:25.571261 7202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f7bc2a5c89b45b5854effb879ee1512" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da565a0436144a6895b0bf8e0dbb1603"
I20260924 20:08:25.571432 7202 raft_consensus.cc:2379] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0f7bc2a5c89b45b5854effb879ee1512 in current term 2: Already voted for candidate da565a0436144a6895b0bf8e0dbb1603 in this term.
I20260924 20:08:25.571388 7384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da565a0436144a6895b0bf8e0dbb1603" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0f7bc2a5c89b45b5854effb879ee1512"
I20260924 20:08:25.571558 7384 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate da565a0436144a6895b0bf8e0dbb1603 in current term 2: Already voted for candidate 0f7bc2a5c89b45b5854effb879ee1512 in this term.
I20260924 20:08:25.571767 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.571549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57084)
0924 20:08:25.571640 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.571701 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 20:08:25.571862 7179 leader_election.cc:334] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475): Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:25.572052 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:25.571837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57098)
0924 20:08:25.571929 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:25.571999 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 20:08:25.572070 7179 leader_election.cc:302] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da565a0436144a6895b0bf8e0dbb1603; no voters: 0f7bc2a5c89b45b5854effb879ee1512, ab3766a2ab094b89b7abed98b7412ac0
W20260924 20:08:25.572158 7362 leader_election.cc:334] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475): Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:25.572201 7898 raft_consensus.cc:2726] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 20:08:25.572273 7362 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f7bc2a5c89b45b5854effb879ee1512; no voters: ab3766a2ab094b89b7abed98b7412ac0, da565a0436144a6895b0bf8e0dbb1603
I20260924 20:08:25.572443 7897 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 20:08:26.051295 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:26.050870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:33635)
0924 20:08:26.051136 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:26.051197 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260924 20:08:26.051332 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:26.050870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:44363)
0924 20:08:26.051190 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:26.051250 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260924 20:08:26.055270 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:26.054868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:38105)
0924 20:08:26.055122 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:26.055182 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260924 20:08:27.052423 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.051959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:46689)
0924 20:08:27.052249 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.052323 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260924 20:08:27.052459 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.052068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:33987)
0924 20:08:27.052325 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.052378 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260924 20:08:27.056358 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.055921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:37645)
0924 20:08:27.056211 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.056270 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
I20260924 20:08:27.088765 7913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:27.088907 7913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:27.089354 7913 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475), da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189:38309)
I20260924 20:08:27.089879 7202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f7bc2a5c89b45b5854effb879ee1512" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da565a0436144a6895b0bf8e0dbb1603" is_pre_election: true
I20260924 20:08:27.090094 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.089600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57356)
0924 20:08:27.089899 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.089959 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260924 20:08:27.090050 7202 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f7bc2a5c89b45b5854effb879ee1512 in term 2.
I20260924 20:08:27.090494 7359 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f7bc2a5c89b45b5854effb879ee1512, da565a0436144a6895b0bf8e0dbb1603; no voters:
I20260924 20:08:27.090709 7913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 20:08:27.090780 7913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:27.090835 7913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:08:27.091120 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.090856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57368)
0924 20:08:27.090974 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.091044 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 20:08:27.091248 7362 leader_election.cc:334] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475): Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:27.092285 7913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:27.092661 7913 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 3 election: Requested vote from peers ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475), da565a0436144a6895b0bf8e0dbb1603 (127.0.115.189:38309)
I20260924 20:08:27.093113 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.092859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57370)
0924 20:08:27.092992 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.093058 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 20:08:27.093146 7202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f7bc2a5c89b45b5854effb879ee1512" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da565a0436144a6895b0bf8e0dbb1603"
I20260924 20:08:27.093312 7202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:08:27.093873 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.093672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57386)
0924 20:08:27.093779 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.093825 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:08:27.093988 7362 leader_election.cc:334] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475): Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
I20260924 20:08:27.094601 7202 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f7bc2a5c89b45b5854effb879ee1512 in term 3.
I20260924 20:08:27.094918 7359 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f7bc2a5c89b45b5854effb879ee1512, da565a0436144a6895b0bf8e0dbb1603; no voters: ab3766a2ab094b89b7abed98b7412ac0
I20260924 20:08:27.095126 7913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:08:27.095371 7913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [term 3 LEADER]: Becoming Leader. State: Replica: 0f7bc2a5c89b45b5854effb879ee1512, State: Running, Role: LEADER
I20260924 20:08:27.095657 7913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } }
I20260924 20:08:27.096869 7916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7bc2a5c89b45b5854effb879ee1512. Latest consensus state: current_term: 3 leader_uuid: "0f7bc2a5c89b45b5854effb879ee1512" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:27.096990 7916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:27.097287 7918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:27.099941 7918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:27.100373 7918 catalog_manager.cc:1259] Loaded cluster ID: 440abd657feb47f5aa8c2bb870b600ef
I20260924 20:08:27.100418 7918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:27.100991 7918 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:27.101408 7918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Loaded TSK: 0
I20260924 20:08:27.102097 7918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:08:27.155457 7374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:27.158730 7374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:27.161937 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.161591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57402)
0924 20:08:27.161781 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.161852 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 20:08:27.161984 7202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [term 3 FOLLOWER]: Refusing update from remote peer 0f7bc2a5c89b45b5854effb879ee1512: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 20:08:27.162520 7916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 20:08:27.162853 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.162566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.0.1:57412)
0924 20:08:27.162723 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.162787 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260924 20:08:27.163087 7362 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 -> Peer ab3766a2ab094b89b7abed98b7412ac0 (127.0.115.190:44475): Couldn't send request to peer ab3766a2ab094b89b7abed98b7412ac0. Status: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 20:08:27.164024 7920 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7bc2a5c89b45b5854effb879ee1512. Latest consensus state: current_term: 3 leader_uuid: "0f7bc2a5c89b45b5854effb879ee1512" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:27.164193 7920 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:08:27.165392 7916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0f7bc2a5c89b45b5854effb879ee1512" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:27.165521 7916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:27.165898 7920 sys_catalog.cc:455] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0f7bc2a5c89b45b5854effb879ee1512" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab3766a2ab094b89b7abed98b7412ac0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44475 } } peers { permanent_uuid: "da565a0436144a6895b0bf8e0dbb1603" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 38309 } attrs { promote: false } } peers { permanent_uuid: "0f7bc2a5c89b45b5854effb879ee1512" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 34931 } attrs { promote: false } } }
I20260924 20:08:27.166075 7920 sys_catalog.cc:458] T 00000000000000000000000000000000 P da565a0436144a6895b0bf8e0dbb1603 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:08:27.177098 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.172506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:44049 (local address 127.0.0.1:59542)
0924 20:08:27.172768 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.172785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:27.172895 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:27.173313 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:27.173317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:27.173336 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:27.173446 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:27.173453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.174584 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.174588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:27.175819 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:27.175832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.176688 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.176692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:27.176777 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:27.176800 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:08:27.176874 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:08:27.176952 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":3}
I20260924 20:08:27.177182 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.172625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.0.115.130:44049)
0924 20:08:27.173033 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:27.173038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:27.173065 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:08:27.173111 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:27.173113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:27.173186 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:27.173286 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:27.173591 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.174439 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.176001 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.176573 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.176698 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:27.176769 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:27.176942 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 20:08:27.177022 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":111,"threads_started":1}
I20260924 20:08:27.177866 7625 tablet_service.cc:1467] Processing CreateTablet for tablet d15ed800be6c46afbb7750294da21ab1 (DEFAULT_TABLE table=test-table [id=b541f2ba1ce748ca8996e15e4173da52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:27.178085 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.172758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:44821 (local address 127.0.0.1:37846)
0924 20:08:27.173683 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.173698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:27.174083 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:27.174449 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:27.174452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:27.174465 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:27.174556 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:27.174562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.175490 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.175495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:27.176486 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:27.176497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.177638 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.177642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:27.177738 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:27.177764 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:08:27.177854 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:08:27.177937 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":90,"threads_started":1}
I20260924 20:08:27.178102 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.173026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.0.115.129:44821)
0924 20:08:27.173324 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:27.173329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:27.173991 (+ 662us) server_negotiation.cc:407] Connection header received
0924 20:08:27.174211 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:27.174213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:27.174261 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:27.174355 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:27.174641 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.175382 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.176625 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.177346 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.177468 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:27.177534 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:27.177891 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 20:08:27.177971 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260924 20:08:27.178196 7625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15ed800be6c46afbb7750294da21ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:27.179473 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.173718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:43971 (local address 127.0.0.1:53906)
0924 20:08:27.173960 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.173974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:27.174065 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:27.174329 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:27.174332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:27.174352 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:27.174447 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:27.174454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.176948 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.176951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:27.178081 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:27.178091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.179042 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.179047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:27.179141 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:27.179157 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 20:08:27.179272 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 20:08:27.179343 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
I20260924 20:08:27.179488 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.173832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.0.115.131:43971)
0924 20:08:27.174084 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:27.174089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:27.174105 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:27.174160 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:27.174163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:27.174213 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:27.174304 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:27.176078 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.176825 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.178226 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.178836 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.178967 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:27.179033 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:27.179290 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 20:08:27.179366 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260924 20:08:27.180071 7762 tablet_service.cc:1467] Processing CreateTablet for tablet d15ed800be6c46afbb7750294da21ab1 (DEFAULT_TABLE table=test-table [id=b541f2ba1ce748ca8996e15e4173da52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:27.180234 7488 tablet_service.cc:1467] Processing CreateTablet for tablet d15ed800be6c46afbb7750294da21ab1 (DEFAULT_TABLE table=test-table [id=b541f2ba1ce748ca8996e15e4173da52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:27.180330 7762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15ed800be6c46afbb7750294da21ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:27.180473 7488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15ed800be6c46afbb7750294da21ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:27.180938 7931 tablet_bootstrap.cc:492] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: Bootstrap starting.
I20260924 20:08:27.181797 7931 tablet_bootstrap.cc:654] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:27.183382 7931 tablet_bootstrap.cc:492] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: No bootstrap required, opened a new log
I20260924 20:08:27.183488 7931 ts_tablet_manager.cc:1397] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:27.183542 7933 tablet_bootstrap.cc:492] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333: Bootstrap starting.
I20260924 20:08:27.183928 7931 raft_consensus.cc:348] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.184106 7931 raft_consensus.cc:374] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:27.184134 7931 raft_consensus.cc:729] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc75e3aa684468382372fa979d43f85, State: Initialized, Role: FOLLOWER
I20260924 20:08:27.184290 7931 consensus_queue.cc:260] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.184365 7933 tablet_bootstrap.cc:654] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:27.185113 7934 tablet_bootstrap.cc:492] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: Bootstrap starting.
I20260924 20:08:27.185405 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.185038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:34959)
0924 20:08:27.185285 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.185339 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260924 20:08:27.185446 7931 ts_tablet_manager.cc:1428] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 20:08:27.185961 7933 tablet_bootstrap.cc:492] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333: No bootstrap required, opened a new log
I20260924 20:08:27.186045 7933 ts_tablet_manager.cc:1397] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:27.186537 7933 raft_consensus.cc:348] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.186685 7933 raft_consensus.cc:374] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:27.186738 7933 raft_consensus.cc:729] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a0ac3b29d3410baa2befb435002333, State: Initialized, Role: FOLLOWER
I20260924 20:08:27.186913 7933 consensus_queue.cc:260] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.187263 7933 ts_tablet_manager.cc:1428] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 20:08:27.188273 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.187994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:37035)
0924 20:08:27.188150 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.188203 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 20:08:27.188293 7692 heartbeater.cc:503] Master 127.0.115.188:34931 was elected leader, sending a full tablet report...
I20260924 20:08:27.188333 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.187994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:42827)
0924 20:08:27.188213 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.188266 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260924 20:08:27.189077 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.188809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:59075)
0924 20:08:27.188942 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.189007 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 20:08:27.189431 7555 heartbeater.cc:503] Master 127.0.115.188:34931 was elected leader, sending a full tablet report...
I20260924 20:08:27.190150 7934 tablet_bootstrap.cc:654] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:27.191385 7934 tablet_bootstrap.cc:492] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: No bootstrap required, opened a new log
I20260924 20:08:27.191488 7934 ts_tablet_manager.cc:1397] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 20:08:27.191900 7934 raft_consensus.cc:348] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.192052 7934 raft_consensus.cc:374] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:27.192078 7934 raft_consensus.cc:729] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720ffa94430342fb91f1b1c247992fce, State: Initialized, Role: FOLLOWER
I20260924 20:08:27.192179 7934 consensus_queue.cc:260] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.192476 7934 ts_tablet_manager.cc:1428] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:27.193053 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.192780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:57721)
0924 20:08:27.192929 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.192987 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 20:08:27.194072 7829 heartbeater.cc:503] Master 127.0.115.188:34931 was elected leader, sending a full tablet report...
I20260924 20:08:27.194178 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.193823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:34747)
0924 20:08:27.194012 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.194064 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260924 20:08:27.318526 7557 tablet.cc:2369] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:27.341976 7939 raft_consensus.cc:482] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:27.342142 7939 raft_consensus.cc:504] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.342716 7939 leader_election.cc:288] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049), 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
I20260924 20:08:27.343235 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15ed800be6c46afbb7750294da21ab1" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc75e3aa684468382372fa979d43f85" is_pre_election: true
I20260924 20:08:27.343276 7508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15ed800be6c46afbb7750294da21ab1" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a0ac3b29d3410baa2befb435002333" is_pre_election: true
I20260924 20:08:27.343418 7645 raft_consensus.cc:2454] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 0.
I20260924 20:08:27.343432 7508 raft_consensus.cc:2454] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 0.
I20260924 20:08:27.343825 7718 leader_election.cc:302] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720ffa94430342fb91f1b1c247992fce, 8fc75e3aa684468382372fa979d43f85; no voters:
I20260924 20:08:27.344039 7939 raft_consensus.cc:2781] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:27.344110 7939 raft_consensus.cc:482] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:27.344184 7939 raft_consensus.cc:3037] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:27.345252 7939 raft_consensus.cc:504] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.345636 7939 leader_election.cc:288] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 election: Requested vote from peers 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049), 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821)
I20260924 20:08:27.345983 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15ed800be6c46afbb7750294da21ab1" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc75e3aa684468382372fa979d43f85"
I20260924 20:08:27.346035 7508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15ed800be6c46afbb7750294da21ab1" candidate_uuid: "720ffa94430342fb91f1b1c247992fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a0ac3b29d3410baa2befb435002333"
I20260924 20:08:27.346108 7645 raft_consensus.cc:3037] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:27.346138 7508 raft_consensus.cc:3037] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:27.347229 7508 raft_consensus.cc:2454] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 1.
I20260924 20:08:27.347218 7645 raft_consensus.cc:2454] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720ffa94430342fb91f1b1c247992fce in term 1.
I20260924 20:08:27.347509 7716 leader_election.cc:302] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a0ac3b29d3410baa2befb435002333, 720ffa94430342fb91f1b1c247992fce; no voters:
I20260924 20:08:27.347661 7939 raft_consensus.cc:2781] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:27.347878 7939 raft_consensus.cc:686] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 1 LEADER]: Becoming Leader. State: Replica: 720ffa94430342fb91f1b1c247992fce, State: Running, Role: LEADER
I20260924 20:08:27.348078 7939 consensus_queue.cc:237] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } }
I20260924 20:08:27.348699 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.348424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:50205)
0924 20:08:27.348584 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.348635 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 20:08:27.349834 7374 catalog_manager.cc:5495] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 720ffa94430342fb91f1b1c247992fce (127.0.115.131). New cstate: current_term: 1 leader_uuid: "720ffa94430342fb91f1b1c247992fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720ffa94430342fb91f1b1c247992fce" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 43971 } health_report { overall_health: HEALTHY } } }
I20260924 20:08:27.366669 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.363540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.0.115.131:43971)
0924 20:08:27.363684 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:27.363689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:27.363705 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:27.363787 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:27.363790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:27.363833 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:27.363912 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:27.364220 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.364745 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.365600 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:27.365828 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:27.365939 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:27.366192 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:27.366266 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:27.366346 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:27.366467 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 20:08:27.366524 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:08:27.369114 7508 raft_consensus.cc:1261] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Refusing update from remote peer 720ffa94430342fb91f1b1c247992fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:27.369174 7645 raft_consensus.cc:1261] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Refusing update from remote peer 720ffa94430342fb91f1b1c247992fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:27.369694 7940 consensus_queue.cc:1035] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc75e3aa684468382372fa979d43f85" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:27.369877 7939 consensus_queue.cc:1035] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a0ac3b29d3410baa2befb435002333" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:27.372740 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.372304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:43133)
0924 20:08:27.372570 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.372631 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260924 20:08:27.373943 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.373244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:48213)
0924 20:08:27.373796 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.373860 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476}
I20260924 20:08:27.377404 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.377036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:36121)
0924 20:08:27.377259 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.377318 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260924 20:08:27.378065 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.377773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.130:39293)
0924 20:08:27.377938 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.377993 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 20:08:27.379454 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.378213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.131:38575)
0924 20:08:27.379329 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.379384 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1047}
I20260924 20:08:27.379465 7375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:27.379637 7375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:27.380667 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.380393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44475 (local address 127.0.115.129:59623)
0924 20:08:27.380539 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.380595 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 20:08:27.383661 7375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f7bc2a5c89b45b5854effb879ee1512: Sending DeleteTablet for 3 replicas of tablet d15ed800be6c46afbb7750294da21ab1
I20260924 20:08:27.384385 7625 tablet_service.cc:1514] Processing DeleteTablet for tablet d15ed800be6c46afbb7750294da21ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:27 UTC) from {username='oryx', principal='oryx/127.0.115.188@KRBTEST.COM'} at 127.0.0.1:59542
I20260924 20:08:27.384462 7488 tablet_service.cc:1514] Processing DeleteTablet for tablet d15ed800be6c46afbb7750294da21ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:27 UTC) from {username='oryx', principal='oryx/127.0.115.188@KRBTEST.COM'} at 127.0.0.1:37846
I20260924 20:08:27.384465 7487 tablet_service.cc:1514] Processing DeleteTablet for tablet d15ed800be6c46afbb7750294da21ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:27 UTC) from {username='oryx', principal='oryx/127.0.115.188@KRBTEST.COM'} at 127.0.0.1:37846
I20260924 20:08:27.384792 7762 tablet_service.cc:1514] Processing DeleteTablet for tablet d15ed800be6c46afbb7750294da21ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:27 UTC) from {username='oryx', principal='oryx/127.0.115.188@KRBTEST.COM'} at 127.0.0.1:53906
I20260924 20:08:27.384909 7952 tablet_replica.cc:330] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: stopping tablet replica
I20260924 20:08:27.384935 7953 tablet_replica.cc:330] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333: stopping tablet replica
I20260924 20:08:27.385040 7952 raft_consensus.cc:2229] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:27.385070 7953 raft_consensus.cc:2229] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:27.385119 7954 tablet_replica.cc:330] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: stopping tablet replica
I20260924 20:08:27.385169 7952 raft_consensus.cc:2258] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:27.385203 7953 raft_consensus.cc:2258] T d15ed800be6c46afbb7750294da21ab1 P 69a0ac3b29d3410baa2befb435002333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:27.385238 7954 raft_consensus.cc:2229] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:27.385543 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 7336
I20260924 20:08:27.386159 7954 raft_consensus.cc:2258] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:27.387169 7954 ts_tablet_manager.cc:1905] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:27.387374 7952 ts_tablet_manager.cc:1905] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:27.389818 7952 ts_tablet_manager.cc:1918] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:27.389922 7952 log.cc:1196] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/d15ed800be6c46afbb7750294da21ab1
I20260924 20:08:27.389997 7954 ts_tablet_manager.cc:1918] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:27.390081 7954 log.cc:1196] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/d15ed800be6c46afbb7750294da21ab1
I20260924 20:08:27.390200 7952 ts_tablet_manager.cc:1939] T d15ed800be6c46afbb7750294da21ab1 P 8fc75e3aa684468382372fa979d43f85: Deleting consensus metadata
I20260924 20:08:27.390455 7954 ts_tablet_manager.cc:1939] T d15ed800be6c46afbb7750294da21ab1 P 720ffa94430342fb91f1b1c247992fce: Deleting consensus metadata
I20260924 20:08:27.390978 7362 catalog_manager.cc:4841] TS 8fc75e3aa684468382372fa979d43f85 (127.0.115.130:44049): tablet d15ed800be6c46afbb7750294da21ab1 (table test-table [id=b541f2ba1ce748ca8996e15e4173da52]) successfully deleted
I20260924 20:08:27.391203 7362 catalog_manager.cc:4841] TS 720ffa94430342fb91f1b1c247992fce (127.0.115.131:43971): tablet d15ed800be6c46afbb7750294da21ab1 (table test-table [id=b541f2ba1ce748ca8996e15e4173da52]) successfully deleted
I20260924 20:08:27.394554 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 7562
I20260924 20:08:27.395334 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:27.394831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:44821 (local address 127.0.0.1:37862)
0924 20:08:27.395150 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:27.395229 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:44821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
W20260924 20:08:27.395516 7360 catalog_manager.cc:4584] TS 69a0ac3b29d3410baa2befb435002333 (127.0.115.129:44821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d15ed800be6c46afbb7750294da21ab1: Network error: Client connection negotiation failed: client connection to 127.0.115.129:44821: connect: Connection refused (error 111)
I20260924 20:08:27.401448 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 7699
I20260924 20:08:27.408501 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 7049
I20260924 20:08:27.414467 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 7156
2026-09-24T20:08:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7063 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:27 dist-test-slave-2pm7 krb5kdc[7957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:27 dist-test-slave-2pm7 krb5kdc[7957](info): set up 2 sockets
Sep 24 20:08:27 dist-test-slave-2pm7 krb5kdc[7957](info): commencing operation
krb5kdc: starting...
W20260924 20:08:28.477373 462 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:28 dist-test-slave-2pm7 krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab.
Entry for principal oryx/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab.
I20260924 20:08:28.601454 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44657
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab
--principal=oryx/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:28.712051 7970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:28.712355 7970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:28.712425 7970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:28.716478 7970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:28.716583 7970 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:08:28.716612 7970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:28.716655 7970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:28.716691 7970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:28.716719 7970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:28.721984 7970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.190.keytab
--principal=oryx/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:28.723101 7970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:28.724170 7970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:28.732688 7978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:28.732728 7976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:28.732872 7970 server_base.cc:1034] running on GCE node
W20260924 20:08:28.732705 7975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:28.733521 7970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:28.734674 7970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:28.735867 7970 hybrid_clock.cc:648] HybridClock initialized: now 1790280508735845 us; error 32 us; skew 500 ppm
Sep 24 20:08:28 dist-test-slave-2pm7 krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280508, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:28.738700 7970 init.cc:377] Logged in from keytab as oryx/127.0.115.190@KRBTEST.COM (short username oryx)
I20260924 20:08:28.739559 7970 webserver.cc:460] Webserver started at http://127.0.115.190:43487/ using document root <none> and password file <none>
I20260924 20:08:28.740145 7970 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:28.740208 7970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:28.740373 7970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:28.742344 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "807de03cc1d547cabeb1efc45c05e932"
format_stamp: "Formatted at 2026-09-24 20:08:28 on dist-test-slave-2pm7"
server_key: "aaa670a55d7d07eb866b8a10faaacc04"
server_key_iv: "61ee425b03530d7b9868105b11b699c5"
server_key_version: "encryptionkey@0"
I20260924 20:08:28.742846 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "807de03cc1d547cabeb1efc45c05e932"
format_stamp: "Formatted at 2026-09-24 20:08:28 on dist-test-slave-2pm7"
server_key: "aaa670a55d7d07eb866b8a10faaacc04"
server_key_iv: "61ee425b03530d7b9868105b11b699c5"
server_key_version: "encryptionkey@0"
I20260924 20:08:28.746536 7970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 20:08:28.748977 7985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:28.750257 7970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:28.750384 7970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "807de03cc1d547cabeb1efc45c05e932"
format_stamp: "Formatted at 2026-09-24 20:08:28 on dist-test-slave-2pm7"
server_key: "aaa670a55d7d07eb866b8a10faaacc04"
server_key_iv: "61ee425b03530d7b9868105b11b699c5"
server_key_version: "encryptionkey@0"
I20260924 20:08:28.750487 7970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:28.777769 7970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:28.780721 7970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:28.780920 7970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:28.789886 7970 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44657
I20260924 20:08:28.789908 8047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44657 every 8 connection(s)
I20260924 20:08:28.790880 7970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:28.793730 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:28.797816 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 7970
I20260924 20:08:28.797951 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:28.798233 462 external_mini_cluster.cc:1428] Setting key 808c5a8f77572dc1ac41a03ad080e62e
I20260924 20:08:28.799602 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Bootstrap starting.
I20260924 20:08:28.802385 8048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:28.803321 8048 log.cc:824] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:28.805392 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: No bootstrap required, opened a new log
Sep 24 20:08:28 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:28.809408 8048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807de03cc1d547cabeb1efc45c05e932" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44657 } }
I20260924 20:08:28.809675 8048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:28.809775 8048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807de03cc1d547cabeb1efc45c05e932, State: Initialized, Role: FOLLOWER
I20260924 20:08:28.810293 8048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807de03cc1d547cabeb1efc45c05e932" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44657 } }
I20260924 20:08:28.810461 8048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:28.810534 8048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:28.810644 8048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:28.811825 8048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807de03cc1d547cabeb1efc45c05e932" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44657 } }
I20260924 20:08:28.812211 8048 leader_election.cc:302] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 807de03cc1d547cabeb1efc45c05e932; no voters:
I20260924 20:08:28.812599 8048 leader_election.cc:288] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:28.812668 8053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:28.813114 8053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [term 1 LEADER]: Becoming Leader. State: Replica: 807de03cc1d547cabeb1efc45c05e932, State: Running, Role: LEADER
I20260924 20:08:28.813476 8053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807de03cc1d547cabeb1efc45c05e932" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44657 } }
I20260924 20:08:28.814086 8048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:28.814117 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:28.799667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.0.115.190:44657)
0924 20:08:28.800150 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:28.800170 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:28.800211 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:08:28.801102 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:28.801126 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:28.801542 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:28.801900 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:28.802616 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:28.804108 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:28.804979 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:28.805675 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:28.805877 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:28.808596 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:28.808602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:28.808616 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:28.808680 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:28.810745 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:28.811183 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:28.811192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:28.811198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:28.811388 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:28.811670 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:28.811676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:28.811681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:28.812268 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:28.812459 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:28.812814 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 20:08:28.813146 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":115,"threads_started":1}
I20260924 20:08:28.814745 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "807de03cc1d547cabeb1efc45c05e932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807de03cc1d547cabeb1efc45c05e932" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44657 } } }
I20260924 20:08:28.814967 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:28.815125 8055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 807de03cc1d547cabeb1efc45c05e932. Latest consensus state: current_term: 1 leader_uuid: "807de03cc1d547cabeb1efc45c05e932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807de03cc1d547cabeb1efc45c05e932" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44657 } } }
I20260924 20:08:28.815294 8055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:28.815411 8062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:28.818686 8062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:28.824361 8062 catalog_manager.cc:1347] Generated new cluster ID: f258a7601da74994a84fdfee4ef0e22b
I20260924 20:08:28.824437 8062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:28.845194 8062 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:28.846235 8062 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:28.855504 8062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Generated new TSK 0
I20260924 20:08:28.856420 8062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab.
Entry for principal oryx/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab.
I20260924 20:08:28.894851 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44657
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab
--principal=oryx/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:29.009367 8074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:29.009680 8074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:29.009765 8074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:29.013823 8074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:29.013921 8074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:29.013990 8074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:08:29.014076 8074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:29.019362 8074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab
--principal=oryx/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:44657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:29.020444 8074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:29.021474 8074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:29.031304 8074 server_base.cc:1034] running on GCE node
W20260924 20:08:29.031256 8079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:29.031296 8080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:29.031255 8082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:29.031874 8074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:29.032553 8074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:29.033794 8074 hybrid_clock.cc:648] HybridClock initialized: now 1790280509033767 us; error 43 us; skew 500 ppm
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:29.036638 8074 init.cc:377] Logged in from keytab as oryx/127.0.115.129@KRBTEST.COM (short username oryx)
I20260924 20:08:29.037456 8074 webserver.cc:460] Webserver started at http://127.0.115.129:44211/ using document root <none> and password file <none>
I20260924 20:08:29.037966 8074 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:29.038022 8074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:29.038266 8074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:29.040349 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "b7905d2f738d46ddbae7ca4d288386c3"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "8154261ccd71a1947c4111c58a045f3c"
server_key_iv: "569dd119b726253176083ddb56b0c50d"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.040877 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "b7905d2f738d46ddbae7ca4d288386c3"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "8154261ccd71a1947c4111c58a045f3c"
server_key_iv: "569dd119b726253176083ddb56b0c50d"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.044693 8074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:08:29.047098 8089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:29.048418 8074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:29.048552 8074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "b7905d2f738d46ddbae7ca4d288386c3"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "8154261ccd71a1947c4111c58a045f3c"
server_key_iv: "569dd119b726253176083ddb56b0c50d"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.048663 8074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:29.076485 8074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:29.079703 8074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:29.079918 8074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:29.081537 8074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:29.081619 8074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:29.081691 8074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:29.081753 8074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:29.092973 8074 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:43597
I20260924 20:08:29.093048 8204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:43597 every 8 connection(s)
I20260924 20:08:29.093890 8074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:29.096164 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.082540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:47569 (local address 127.0.115.190:44657)
0924 20:08:29.082679 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.082682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.083821 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 20:08:29.084950 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.084956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.085025 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.085089 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.086923 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.087866 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.089273 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.089544 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.089669 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.092370 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.092374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.092377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.092416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.094011 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.094576 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.094580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.094582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.094666 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.095063 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.095066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.095068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.095277 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.095370 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.095837 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.096027 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:08:29.096920 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.083116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44657 (local address 127.0.115.129:47569)
0924 20:08:29.083664 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.083710 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.084688 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.085618 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.085636 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.086052 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:29.086739 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.086758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.088062 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.088068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.089145 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.089155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.089724 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.089728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.089905 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.090525 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:29.090542 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:29.092199 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:29.094115 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.094121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.094142 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.094450 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.094765 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.094768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.094770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.094943 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.095377 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:29.095382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:29.095595 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.095767 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.096079 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260924 20:08:29.098191 8205 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44657
I20260924 20:08:29.098325 8205 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:29.098945 8205 heartbeater.cc:511] Master 127.0.115.190:44657 requested a full tablet report, sending...
I20260924 20:08:29.100724 8002 ts_manager.cc:194] Registered new tserver with Master: b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597)
I20260924 20:08:29.102239 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 8074
I20260924 20:08:29.102288 8002 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:47569
I20260924 20:08:29.102370 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:29.102733 462 external_mini_cluster.cc:1428] Setting key ab7e0c36e75b8bbe566b3befa02e7516
I20260924 20:08:29.108525 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.100628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:60673 (local address 127.0.115.190:44657)
0924 20:08:29.100764 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.100768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.100941 (+ 173us) server_negotiation.cc:407] Connection header received
0924 20:08:29.101053 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.101056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.101098 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.101179 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.101579 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.102334 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.103485 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.103809 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.103911 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.105147 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.105150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.105153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.105180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.106863 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.107287 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.107292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.107295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.107385 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.107710 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.107712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.107713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.107886 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.108029 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.108322 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.108414 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:08:29.108703 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.100551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44657 (local address 127.0.115.129:60673)
0924 20:08:29.100836 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.100848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.100962 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.101213 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.101215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.101373 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:29.101470 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.101475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.102460 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.102464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.103330 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.103341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.103938 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.103964 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.104054 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.104493 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:29.104507 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:29.105006 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:29.106977 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.106979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.106981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.107188 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.107506 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.107512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.107517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.107616 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.108012 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:29.108019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:29.108153 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.108272 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.108435 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab.
Entry for principal oryx/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab.
I20260924 20:08:29.131423 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44657
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab
--principal=oryx/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:29.242386 8214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:29.242672 8214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:29.242748 8214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:29.246922 8214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:29.247004 8214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:29.247074 8214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:08:29.247160 8214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:29.252455 8214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab
--principal=oryx/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:44657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:29.253532 8214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:29.254621 8214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:29.264338 8222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:29.264340 8219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:29.264405 8214 server_base.cc:1034] running on GCE node
W20260924 20:08:29.264340 8220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:29.264964 8214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:29.265571 8214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:29.266757 8214 hybrid_clock.cc:648] HybridClock initialized: now 1790280509266742 us; error 24 us; skew 500 ppm
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:29.269457 8214 init.cc:377] Logged in from keytab as oryx/127.0.115.130@KRBTEST.COM (short username oryx)
I20260924 20:08:29.270304 8214 webserver.cc:460] Webserver started at http://127.0.115.130:38203/ using document root <none> and password file <none>
I20260924 20:08:29.270804 8214 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:29.270861 8214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:29.271020 8214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:29.273044 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "2d10ac6656c743d2b39587658c88728a"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "57a3c257dc6b0f77a06a8471e3d8f9e4"
server_key_iv: "821883cc902be478590a1c9ffc061430"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.273590 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "2d10ac6656c743d2b39587658c88728a"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "57a3c257dc6b0f77a06a8471e3d8f9e4"
server_key_iv: "821883cc902be478590a1c9ffc061430"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.277269 8214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:08:29.279666 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:29.280931 8214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:29.281036 8214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "2d10ac6656c743d2b39587658c88728a"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "57a3c257dc6b0f77a06a8471e3d8f9e4"
server_key_iv: "821883cc902be478590a1c9ffc061430"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.281123 8214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:29.319919 8214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:29.323555 8214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:29.323727 8214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:29.325271 8214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:29.325325 8214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:29.325389 8214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:29.325429 8214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:29.336958 8214 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:40167
I20260924 20:08:29.337016 8344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:40167 every 8 connection(s)
I20260924 20:08:29.338136 8214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:29.340337 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.326435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:55521 (local address 127.0.115.190:44657)
0924 20:08:29.326598 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.326604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.327509 (+ 905us) server_negotiation.cc:407] Connection header received
0924 20:08:29.328908 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.328912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.328978 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.329085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.330574 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.331181 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.332621 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.332880 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.333008 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.336118 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.336121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.336124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.336151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.337909 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.338615 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.338619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.338621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.338733 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.339151 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.339153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.339155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.339352 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.339434 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.340039 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.340177 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:08:29.341099 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.326854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44657 (local address 127.0.115.130:55521)
0924 20:08:29.327347 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.327402 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.328667 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.329289 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.329297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.329859 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:29.330404 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.330419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.331323 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.331327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.332484 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.332494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.333077 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.333083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.333254 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.333988 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:29.334005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:29.335931 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:29.338019 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.338025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.338041 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.338388 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.338856 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.338859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.338861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.339026 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.339458 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:29.339464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:29.339712 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.339866 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.340195 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":152,"threads_started":1}
I20260924 20:08:29.342319 8345 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44657
I20260924 20:08:29.342479 8345 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:29.343019 8345 heartbeater.cc:511] Master 127.0.115.190:44657 requested a full tablet report, sending...
I20260924 20:08:29.344146 8001 ts_manager.cc:194] Registered new tserver with Master: 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167)
I20260924 20:08:29.344951 8001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:55521
I20260924 20:08:29.347916 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 8214
I20260924 20:08:29.348106 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:29.348438 462 external_mini_cluster.cc:1428] Setting key 7d89e87df641255d8a40ae5bc9f2d3ce
I20260924 20:08:29.352375 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.344610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44657 (local address 127.0.115.130:46221)
0924 20:08:29.345029 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.345045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.345139 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.345367 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.345370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.345594 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:29.345727 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.345735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.346469 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.346472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.347208 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.347217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.347660 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.347662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.347732 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.348330 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:29.348341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:29.348705 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:29.350495 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.350497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.350499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.350739 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.351035 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.351037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.351038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.351112 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.351692 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:29.351699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:29.351817 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.351935 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.352119 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260924 20:08:29.352959 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.344969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:46221 (local address 127.0.115.190:44657)
0924 20:08:29.345143 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.345147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.345162 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:29.345224 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.345227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.345273 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.345347 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.345844 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.346338 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.347318 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.347525 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.347603 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.348847 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.348850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.348852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.348874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.350377 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.350838 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.350845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.350850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.350943 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.351198 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.351204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.351208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.351514 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.351622 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.352618 (+ 996us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.352755 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
WARNING: no policy specified for oryx/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab.
Entry for principal oryx/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab.
I20260924 20:08:29.378973 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44657
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab
--principal=oryx/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:29.493747 8354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:29.494122 8354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:29.494195 8354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:29.499620 8354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:29.499715 8354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:29.499833 8354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:08:29.499960 8354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:29.507398 8354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab
--principal=oryx/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.115.190:44657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:29.508680 8354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:29.509804 8354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:29.520036 8362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:29.520022 8360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:29.520022 8359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:29.520515 8354 server_base.cc:1034] running on GCE node
I20260924 20:08:29.520931 8354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:29.521607 8354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:29.522809 8354 hybrid_clock.cc:648] HybridClock initialized: now 1790280509522781 us; error 54 us; skew 500 ppm
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:29.525604 8354 init.cc:377] Logged in from keytab as oryx/127.0.115.131@KRBTEST.COM (short username oryx)
I20260924 20:08:29.526480 8354 webserver.cc:460] Webserver started at http://127.0.115.131:38589/ using document root <none> and password file <none>
I20260924 20:08:29.527055 8354 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:29.527143 8354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:29.527372 8354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:29.529495 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "6c29aeb950ca4aa9ab4b55634fcb7473"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "e62a5f7fdb0360e3b9ed973559d21413"
server_key_iv: "f2619bc5e0ef6c0cc06fe6b41c6f3570"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.530095 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "6c29aeb950ca4aa9ab4b55634fcb7473"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "e62a5f7fdb0360e3b9ed973559d21413"
server_key_iv: "f2619bc5e0ef6c0cc06fe6b41c6f3570"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.534070 8354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:29.536579 8369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:29.537925 8354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:29.538069 8354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "6c29aeb950ca4aa9ab4b55634fcb7473"
format_stamp: "Formatted at 2026-09-24 20:08:29 on dist-test-slave-2pm7"
server_key: "e62a5f7fdb0360e3b9ed973559d21413"
server_key_iv: "f2619bc5e0ef6c0cc06fe6b41c6f3570"
server_key_version: "encryptionkey@0"
I20260924 20:08:29.538189 8354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:29.559475 8354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:29.562927 8354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:29.563082 8354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:29.564630 8354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:29.564682 8354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:29.564723 8354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:29.564739 8354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:29.575842 8354 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:38673
I20260924 20:08:29.575929 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:38673 every 8 connection(s)
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:29.576857 8354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:29.580658 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.565918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:41617 (local address 127.0.115.190:44657)
0924 20:08:29.566056 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.566060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.567116 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 20:08:29.568467 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.568470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.568532 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.568649 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.570404 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.571433 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.573310 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.573702 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.573828 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.576889 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.576894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.576897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.576937 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.578556 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.579084 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.579086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.579088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.579163 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.579553 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.579555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.579556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.579730 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.579821 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.580438 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.580540 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:08:29.581542 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.566439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44657 (local address 127.0.115.131:41617)
0924 20:08:29.566945 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.566989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.568247 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.568865 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.568874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.569410 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:29.570197 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.570220 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.571571 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.571578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.573167 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.573179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.573872 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.573877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.574082 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.574816 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:29.574835 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:29.576690 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:29.578679 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.578686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.578700 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.578974 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.579291 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.579294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.579296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.579454 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.579843 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:29.579848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:29.580201 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.580376 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.580662 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":139,"threads_started":1}
I20260924 20:08:29.583020 8485 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44657
I20260924 20:08:29.583189 8485 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:29.583735 8485 heartbeater.cc:511] Master 127.0.115.190:44657 requested a full tablet report, sending...
I20260924 20:08:29.584841 8001 ts_manager.cc:194] Registered new tserver with Master: 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:29.585422 8001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:41617
I20260924 20:08:29.585786 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 8354
I20260924 20:08:29.585922 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:29.586288 462 external_mini_cluster.cc:1428] Setting key cc007555f1294ac993c7bd1f73f83e39
I20260924 20:08:29.589316 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:08:29.595295 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.586318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:52503 (local address 127.0.115.190:44657)
0924 20:08:29.586430 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.586436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.586450 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:29.586487 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.586489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.586530 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.586613 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.587140 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.587661 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.588580 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.588791 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.588871 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.591554 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.591560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.591565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.591598 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.593306 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.593900 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.593910 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.593911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.593995 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.594340 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.594344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.594346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.594551 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.594673 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.594974 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.595111 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 20:08:29.595309 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.585929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44657 (local address 127.0.115.131:52503)
0924 20:08:29.586194 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.586206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.586317 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.586654 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.586656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.586885 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:29.587030 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.587040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.587793 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.587796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.588445 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.588475 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.588903 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.588907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.589009 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.591068 (+ 2059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:29.591080 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:29.591396 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:29.593455 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.593459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.593461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.593767 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.594122 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.594126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.594130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.594219 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.594661 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:29.594664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:29.594789 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.594909 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.595111 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":116,"threads_started":1}
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:08:29.614378 8051 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:29.608105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.0.115.190:44657)
0924 20:08:29.608243 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.608247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.608259 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:08:29.608298 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.608301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.608338 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.608410 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.609208 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.609734 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.610537 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.610776 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.610859 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.614221 (+ 3362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53738: BlockingRecv error: recv got EOF from 127.0.0.1:53738 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280509, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:29.623737 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.615959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.0.115.190:44657)
0924 20:08:29.616100 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.616122 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.616275 (+ 153us) server_negotiation.cc:407] Connection header received
0924 20:08:29.616423 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.616426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.616475 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.616556 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.617127 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.617631 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.618424 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.618642 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.618751 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.621078 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.621081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.621084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.621107 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.622351 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.622750 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.622753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.622754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.622824 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.623049 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.623051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.623052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.623216 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.623316 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.623511 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.623601 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:08:29.626660 8001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:29.629815 8001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:29.644932 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.638346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.0.115.130:40167)
0924 20:08:29.638700 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.638712 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.639583 (+ 871us) server_negotiation.cc:407] Connection header received
0924 20:08:29.639679 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.639686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.639907 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.640086 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:29.640493 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.641747 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.643167 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.644030 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.644140 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.644300 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.644677 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.644773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":170,"threads_started":1}
I20260924 20:08:29.644918 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.638166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:40167 (local address 127.0.0.1:46874)
0924 20:08:29.639249 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.639298 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.639436 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.640100 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.640106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.640139 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:29.640341 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.640352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.641871 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.641875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.643031 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.643041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.644362 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.644369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.644483 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.644514 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.644640 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.644763 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":118,"spinlock_wait_cycles":2816,"thread_start_us":91,"threads_started":1}
I20260924 20:08:29.645551 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.638557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:38673 (local address 127.0.0.1:55990)
0924 20:08:29.639488 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.639502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.639608 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.640103 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.640106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.640143 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:29.640318 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.640324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.641684 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.641691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.643993 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.644004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.645142 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.645146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.645252 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.645278 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.645338 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.645405 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":1068,"spinlock_wait_cycles":4992,"thread_start_us":90,"threads_started":1}
I20260924 20:08:29.646297 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.639151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.115.131:38673)
0924 20:08:29.639516 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.639534 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.639565 (+ 31us) server_negotiation.cc:407] Connection header received
0924 20:08:29.639754 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.639758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.639906 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.640488 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:29.640626 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.641398 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.644121 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.644863 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.645821 (+ 958us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.645991 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.646049 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.646151 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
I20260924 20:08:29.646708 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.638877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:43597 (local address 127.0.0.1:51420)
0924 20:08:29.639806 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.639818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.639930 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.640431 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.640433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.640452 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:29.640521 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.640525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.641880 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.641883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.644285 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.644296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.646315 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.646320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.646419 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.646447 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.646518 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.646588 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":1147,"thread_start_us":81,"threads_started":1}
I20260924 20:08:29.647183 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.639118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.0.115.129:43597)
0924 20:08:29.639755 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.639762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.639908 (+ 146us) server_negotiation.cc:407] Connection header received
0924 20:08:29.640092 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.640098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.640285 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.640440 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:29.640612 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.641732 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.644410 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.646007 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.646107 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.646246 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.646972 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.647075 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":92,"threads_started":1}
I20260924 20:08:29.648377 8279 tablet_service.cc:1467] Processing CreateTablet for tablet bdc5aaca3fcc461787da3d77bc9117bb (DEFAULT_TABLE table=test-table [id=78ba46177b944525b7a382d98bcc3f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:29.648900 8419 tablet_service.cc:1467] Processing CreateTablet for tablet bdc5aaca3fcc461787da3d77bc9117bb (DEFAULT_TABLE table=test-table [id=78ba46177b944525b7a382d98bcc3f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:29.649524 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc5aaca3fcc461787da3d77bc9117bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.649327 8139 tablet_service.cc:1467] Processing CreateTablet for tablet bdc5aaca3fcc461787da3d77bc9117bb (DEFAULT_TABLE table=test-table [id=78ba46177b944525b7a382d98bcc3f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:29.649856 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc5aaca3fcc461787da3d77bc9117bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.650175 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc5aaca3fcc461787da3d77bc9117bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.656168 8513 tablet_bootstrap.cc:492] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Bootstrap starting.
I20260924 20:08:29.657904 8515 tablet_bootstrap.cc:492] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Bootstrap starting.
I20260924 20:08:29.658527 8513 tablet_bootstrap.cc:654] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.659271 8513 log.cc:824] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:29.659754 8515 tablet_bootstrap.cc:654] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.660027 8514 tablet_bootstrap.cc:492] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Bootstrap starting.
I20260924 20:08:29.660524 8515 log.cc:824] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:29.661832 8514 tablet_bootstrap.cc:654] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.662431 8515 tablet_bootstrap.cc:492] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: No bootstrap required, opened a new log
I20260924 20:08:29.662505 8514 log.cc:824] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:29.662650 8515 ts_tablet_manager.cc:1397] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:08:29.663920 8513 tablet_bootstrap.cc:492] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: No bootstrap required, opened a new log
I20260924 20:08:29.664139 8513 ts_tablet_manager.cc:1397] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 20:08:29.664364 8514 tablet_bootstrap.cc:492] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: No bootstrap required, opened a new log
I20260924 20:08:29.664534 8514 ts_tablet_manager.cc:1397] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:08:29.666529 8515 raft_consensus.cc:348] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.666805 8515 raft_consensus.cc:374] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.666858 8515 raft_consensus.cc:729] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10ac6656c743d2b39587658c88728a, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.667338 8515 consensus_queue.cc:260] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.668051 8345 heartbeater.cc:503] Master 127.0.115.190:44657 was elected leader, sending a full tablet report...
I20260924 20:08:29.668370 8515 ts_tablet_manager.cc:1428] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:08:29.668365 8514 raft_consensus.cc:348] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.668608 8514 raft_consensus.cc:374] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.668669 8514 raft_consensus.cc:729] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c29aeb950ca4aa9ab4b55634fcb7473, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.669054 8514 consensus_queue.cc:260] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.669231 8513 raft_consensus.cc:348] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.669477 8513 raft_consensus.cc:374] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.669533 8513 raft_consensus.cc:729] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7905d2f738d46ddbae7ca4d288386c3, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.669931 8513 consensus_queue.cc:260] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.669986 8514 ts_tablet_manager.cc:1428] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:08:29.670337 8485 heartbeater.cc:503] Master 127.0.115.190:44657 was elected leader, sending a full tablet report...
I20260924 20:08:29.670578 8205 heartbeater.cc:503] Master 127.0.115.190:44657 was elected leader, sending a full tablet report...
I20260924 20:08:29.671869 8513 ts_tablet_manager.cc:1428] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260924 20:08:29.681484 8519 raft_consensus.cc:482] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:29.681627 8519 raft_consensus.cc:504] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.682770 8519 leader_election.cc:288] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:29.687263 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.683262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:48083 (local address 127.0.115.129:43597)
0924 20:08:29.683377 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.683381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.683503 (+ 122us) server_negotiation.cc:407] Connection header received
0924 20:08:29.683690 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.683692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.683730 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.683815 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:29.684160 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.684781 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.686230 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.686707 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.686842 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.686899 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.687075 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.687143 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:08:29.687286 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.683026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:43597 (local address 127.0.115.130:48083)
0924 20:08:29.683419 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.683430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.683549 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.683826 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.683829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.683845 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:29.684017 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.684030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.684920 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.684923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.686100 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.686110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.686835 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.686838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.686944 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.686972 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.687059 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.687127 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":57,"thread_start_us":117,"threads_started":1}
I20260924 20:08:29.687505 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.683092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:46065 (local address 127.0.115.131:38673)
0924 20:08:29.683198 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.683203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.683219 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:29.683426 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.683428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.683474 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.683637 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:29.683898 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.684699 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.686397 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.686893 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.687039 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.687090 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.687312 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.687386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 20:08:29.687528 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.683015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:38673 (local address 127.0.115.130:46065)
0924 20:08:29.683151 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.683172 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.683269 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.683584 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.683588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.683609 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:29.683762 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.683770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.684841 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.684844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.686272 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.686282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.687030 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.687034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.687165 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.687195 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.687295 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.687372 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 20:08:29.687867 8159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc5aaca3fcc461787da3d77bc9117bb" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7905d2f738d46ddbae7ca4d288386c3" is_pre_election: true
I20260924 20:08:29.688166 8159 raft_consensus.cc:2454] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 0.
I20260924 20:08:29.688498 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc5aaca3fcc461787da3d77bc9117bb" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" is_pre_election: true
I20260924 20:08:29.688663 8230 leader_election.cc:302] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10ac6656c743d2b39587658c88728a, b7905d2f738d46ddbae7ca4d288386c3; no voters:
I20260924 20:08:29.688768 8439 raft_consensus.cc:2454] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 0.
I20260924 20:08:29.688928 8519 raft_consensus.cc:2781] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:29.689061 8519 raft_consensus.cc:482] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:29.689147 8519 raft_consensus.cc:3037] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:29.690256 8519 raft_consensus.cc:504] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.690632 8519 leader_election.cc:288] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 election: Requested vote from peers b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:29.690990 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc5aaca3fcc461787da3d77bc9117bb" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473"
I20260924 20:08:29.691028 8159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc5aaca3fcc461787da3d77bc9117bb" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7905d2f738d46ddbae7ca4d288386c3"
I20260924 20:08:29.691123 8439 raft_consensus.cc:3037] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:29.691148 8159 raft_consensus.cc:3037] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:29.692557 8159 raft_consensus.cc:2454] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 1.
I20260924 20:08:29.692557 8439 raft_consensus.cc:2454] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 1.
I20260924 20:08:29.692893 8233 leader_election.cc:302] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10ac6656c743d2b39587658c88728a, 6c29aeb950ca4aa9ab4b55634fcb7473; no voters:
I20260924 20:08:29.693120 8519 raft_consensus.cc:2781] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:29.693353 8519 raft_consensus.cc:686] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 1 LEADER]: Becoming Leader. State: Replica: 2d10ac6656c743d2b39587658c88728a, State: Running, Role: LEADER
I20260924 20:08:29.693740 8519 consensus_queue.cc:237] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.697053 8001 catalog_manager.cc:5495] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d10ac6656c743d2b39587658c88728a (127.0.115.130). New cstate: current_term: 1 leader_uuid: "2d10ac6656c743d2b39587658c88728a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:29.723169 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.719558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.0.115.130:40167)
0924 20:08:29.719719 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.719725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.719740 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:29.719832 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.719834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.719888 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.720046 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:29.720295 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.720820 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.721676 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.721908 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.722023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.722313 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:29.722773 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:29.722902 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.722960 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.723020 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:08:29.729308 8159 raft_consensus.cc:1261] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Refusing update from remote peer 2d10ac6656c743d2b39587658c88728a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:29.729312 8439 raft_consensus.cc:1261] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Refusing update from remote peer 2d10ac6656c743d2b39587658c88728a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:29.730082 8519 consensus_queue.cc:1035] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:29.730343 8523 consensus_queue.cc:1035] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:29.749266 8001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:29.749467 8001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:29.751895 8001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Sending DeleteTablet for 3 replicas of tablet bdc5aaca3fcc461787da3d77bc9117bb
I20260924 20:08:29.752821 8279 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc5aaca3fcc461787da3d77bc9117bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:29 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:46874
I20260924 20:08:29.752816 8419 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc5aaca3fcc461787da3d77bc9117bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:29 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260924 20:08:29.753021 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc5aaca3fcc461787da3d77bc9117bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:29 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:51420
I20260924 20:08:29.753600 8538 tablet_replica.cc:330] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: stopping tablet replica
I20260924 20:08:29.753801 8536 tablet_replica.cc:330] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: stopping tablet replica
I20260924 20:08:29.753973 8538 raft_consensus.cc:2229] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:29.754027 8536 raft_consensus.cc:2229] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:29.754024 8001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:08:29.754284 8538 raft_consensus.cc:2258] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:29.754384 8536 raft_consensus.cc:2258] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:08:29.754431 8001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:29.754752 8537 tablet_replica.cc:330] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: stopping tablet replica
I20260924 20:08:29.755061 8537 raft_consensus.cc:2229] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:29.756039 8538 ts_tablet_manager.cc:1905] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:29.755880 8537 raft_consensus.cc:2258] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:29.759485 8538 ts_tablet_manager.cc:1918] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:29.759605 8538 log.cc:1196] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/bdc5aaca3fcc461787da3d77bc9117bb
I20260924 20:08:29.759972 8538 ts_tablet_manager.cc:1939] T bdc5aaca3fcc461787da3d77bc9117bb P b7905d2f738d46ddbae7ca4d288386c3: Deleting consensus metadata
I20260924 20:08:29.760185 8279 tablet_service.cc:1467] Processing CreateTablet for tablet bfc1a70c387340a6a120eb9d1e7ee15b (DEFAULT_TABLE table=test-table [id=68b7bc8c485144e29d18ef1a30be1f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:29.760293 8139 tablet_service.cc:1467] Processing CreateTablet for tablet bfc1a70c387340a6a120eb9d1e7ee15b (DEFAULT_TABLE table=test-table [id=68b7bc8c485144e29d18ef1a30be1f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:29.760474 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc1a70c387340a6a120eb9d1e7ee15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.760560 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc1a70c387340a6a120eb9d1e7ee15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.761322 7986 catalog_manager.cc:4841] TS b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597): tablet bdc5aaca3fcc461787da3d77bc9117bb (table test-table [id=78ba46177b944525b7a382d98bcc3f30]) successfully deleted
I20260924 20:08:29.762683 8515 tablet_bootstrap.cc:492] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Bootstrap starting.
I20260924 20:08:29.762894 8513 tablet_bootstrap.cc:492] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Bootstrap starting.
I20260924 20:08:29.763746 8515 tablet_bootstrap.cc:654] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.763984 8513 tablet_bootstrap.cc:654] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.764580 8536 ts_tablet_manager.cc:1905] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:29.764566 8419 tablet_service.cc:1467] Processing CreateTablet for tablet bfc1a70c387340a6a120eb9d1e7ee15b (DEFAULT_TABLE table=test-table [id=68b7bc8c485144e29d18ef1a30be1f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:29.764829 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc1a70c387340a6a120eb9d1e7ee15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.767141 8536 ts_tablet_manager.cc:1918] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:29.767238 8536 log.cc:1196] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/bdc5aaca3fcc461787da3d77bc9117bb
I20260924 20:08:29.767290 8514 tablet_bootstrap.cc:492] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Bootstrap starting.
I20260924 20:08:29.767582 8536 ts_tablet_manager.cc:1939] T bdc5aaca3fcc461787da3d77bc9117bb P 2d10ac6656c743d2b39587658c88728a: Deleting consensus metadata
I20260924 20:08:29.768249 8515 tablet_bootstrap.cc:492] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: No bootstrap required, opened a new log
I20260924 20:08:29.768338 8515 ts_tablet_manager.cc:1397] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260924 20:08:29.768337 8514 tablet_bootstrap.cc:654] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.768514 8537 ts_tablet_manager.cc:1905] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:29.768810 8515 raft_consensus.cc:348] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.768946 8515 raft_consensus.cc:374] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.768996 8515 raft_consensus.cc:729] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10ac6656c743d2b39587658c88728a, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.769126 8515 consensus_queue.cc:260] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.769403 8515 ts_tablet_manager.cc:1428] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:08:29.770860 8513 tablet_bootstrap.cc:492] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: No bootstrap required, opened a new log
I20260924 20:08:29.770965 8513 ts_tablet_manager.cc:1397] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260924 20:08:29.771113 8537 ts_tablet_manager.cc:1918] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:29.771209 8537 log.cc:1196] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/bdc5aaca3fcc461787da3d77bc9117bb
I20260924 20:08:29.771534 8537 ts_tablet_manager.cc:1939] T bdc5aaca3fcc461787da3d77bc9117bb P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting consensus metadata
I20260924 20:08:29.771517 8513 raft_consensus.cc:348] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.771648 8513 raft_consensus.cc:374] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.771694 8513 raft_consensus.cc:729] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7905d2f738d46ddbae7ca4d288386c3, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.771852 8513 consensus_queue.cc:260] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.772140 8513 ts_tablet_manager.cc:1428] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 20:08:29.774513 7986 catalog_manager.cc:4841] TS 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167): tablet bdc5aaca3fcc461787da3d77bc9117bb (table test-table [id=78ba46177b944525b7a382d98bcc3f30]) successfully deleted
I20260924 20:08:29.775154 7989 catalog_manager.cc:4841] TS 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673): tablet bdc5aaca3fcc461787da3d77bc9117bb (table test-table [id=78ba46177b944525b7a382d98bcc3f30]) successfully deleted
I20260924 20:08:29.775743 8514 tablet_bootstrap.cc:492] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: No bootstrap required, opened a new log
I20260924 20:08:29.775825 8514 ts_tablet_manager.cc:1397] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 20:08:29.776286 8514 raft_consensus.cc:348] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.776382 8514 raft_consensus.cc:374] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.776403 8514 raft_consensus.cc:729] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c29aeb950ca4aa9ab4b55634fcb7473, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.776553 8514 consensus_queue.cc:260] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.776826 8514 ts_tablet_manager.cc:1428] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260924 20:08:29.827975 8486 tablet.cc:2369] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:08:29.839412 8346 tablet.cc:2369] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:08:29.845646 8206 tablet.cc:2369] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:08:29.850044 8531 raft_consensus.cc:482] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:29.850193 8531 raft_consensus.cc:504] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.850669 8531 leader_election.cc:288] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:29.851150 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc1a70c387340a6a120eb9d1e7ee15b" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" is_pre_election: true
I20260924 20:08:29.851143 8159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc1a70c387340a6a120eb9d1e7ee15b" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7905d2f738d46ddbae7ca4d288386c3" is_pre_election: true
I20260924 20:08:29.851310 8159 raft_consensus.cc:2454] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 0.
I20260924 20:08:29.851305 8439 raft_consensus.cc:2454] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 0.
I20260924 20:08:29.851634 8233 leader_election.cc:302] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10ac6656c743d2b39587658c88728a, 6c29aeb950ca4aa9ab4b55634fcb7473; no voters:
I20260924 20:08:29.851796 8531 raft_consensus.cc:2781] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:29.851858 8531 raft_consensus.cc:482] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:29.851903 8531 raft_consensus.cc:3037] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:29.853017 8531 raft_consensus.cc:504] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.853421 8531 leader_election.cc:288] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 election: Requested vote from peers b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:29.853828 8159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc1a70c387340a6a120eb9d1e7ee15b" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7905d2f738d46ddbae7ca4d288386c3"
I20260924 20:08:29.853828 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc1a70c387340a6a120eb9d1e7ee15b" candidate_uuid: "2d10ac6656c743d2b39587658c88728a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473"
I20260924 20:08:29.853960 8159 raft_consensus.cc:3037] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:29.853960 8439 raft_consensus.cc:3037] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:29.855137 8159 raft_consensus.cc:2454] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 1.
I20260924 20:08:29.855137 8439 raft_consensus.cc:2454] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10ac6656c743d2b39587658c88728a in term 1.
I20260924 20:08:29.855459 8230 leader_election.cc:302] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10ac6656c743d2b39587658c88728a, b7905d2f738d46ddbae7ca4d288386c3; no voters:
I20260924 20:08:29.855669 8531 raft_consensus.cc:2781] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:29.855758 8531 raft_consensus.cc:686] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 1 LEADER]: Becoming Leader. State: Replica: 2d10ac6656c743d2b39587658c88728a, State: Running, Role: LEADER
I20260924 20:08:29.855906 8531 consensus_queue.cc:237] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.857713 8002 catalog_manager.cc:5495] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d10ac6656c743d2b39587658c88728a (127.0.115.130). New cstate: current_term: 1 leader_uuid: "2d10ac6656c743d2b39587658c88728a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:08:29 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280508, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.190@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:29.895808 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.886039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44657 (local address 127.0.0.1:53754)
0924 20:08:29.886358 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:29.886376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:29.886508 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:29.886841 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:29.886844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:29.887257 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:29.887370 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.887376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.888450 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.888453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.889080 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:29.889089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.889692 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.889695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:29.889765 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.890431 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:29.890445 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:29.891923 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:29.894061 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.894070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.894089 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.894318 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.894669 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:29.894672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:29.894674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:29.894765 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:29.895262 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:29.895268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:29.895383 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 20:08:29.895505 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 20:08:29.895656 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":117,"threads_started":1}
I20260924 20:08:29.895921 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:29.886154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.0.115.190:44657)
0924 20:08:29.886318 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:29.886323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:29.886469 (+ 146us) server_negotiation.cc:407] Connection header received
0924 20:08:29.886640 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:29.886645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:29.886701 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:29.886808 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:29.887505 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.888335 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.889220 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:29.889556 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:29.889672 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:29.892129 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:29.892133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:29.892136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:29.892171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:29.893932 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.894428 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.894432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.894434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.894527 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:29.894876 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:29.894879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:29.894881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:29.895113 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:29.895235 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:29.895593 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 20:08:29.895725 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:08:29.900269 8002 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:53754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:08:29.901032 8002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:29.906315 8139 tablet_service.cc:1467] Processing CreateTablet for tablet bea2c24f9d544886902e8e870273bc60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=228471e43be14307859472ba5adb6903]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:08:29.906315 8279 tablet_service.cc:1467] Processing CreateTablet for tablet bea2c24f9d544886902e8e870273bc60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=228471e43be14307859472ba5adb6903]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:08:29.906503 8419 tablet_service.cc:1467] Processing CreateTablet for tablet bea2c24f9d544886902e8e870273bc60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=228471e43be14307859472ba5adb6903]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:08:29.906734 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea2c24f9d544886902e8e870273bc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.906734 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea2c24f9d544886902e8e870273bc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.906893 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea2c24f9d544886902e8e870273bc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:29.909569 8515 tablet_bootstrap.cc:492] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: Bootstrap starting.
I20260924 20:08:29.910049 8514 tablet_bootstrap.cc:492] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: Bootstrap starting.
I20260924 20:08:29.910535 8515 tablet_bootstrap.cc:654] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.911029 8514 tablet_bootstrap.cc:654] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.911098 8513 tablet_bootstrap.cc:492] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: Bootstrap starting.
I20260924 20:08:29.912110 8513 tablet_bootstrap.cc:654] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:29.912606 8514 tablet_bootstrap.cc:492] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: No bootstrap required, opened a new log
I20260924 20:08:29.912695 8514 ts_tablet_manager.cc:1397] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 20:08:29.913170 8515 tablet_bootstrap.cc:492] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: No bootstrap required, opened a new log
I20260924 20:08:29.913280 8515 ts_tablet_manager.cc:1397] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 20:08:29.913204 8514 raft_consensus.cc:348] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.913378 8514 raft_consensus.cc:374] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.913414 8514 raft_consensus.cc:729] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c29aeb950ca4aa9ab4b55634fcb7473, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.913700 8515 raft_consensus.cc:348] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.913838 8515 raft_consensus.cc:374] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.913873 8515 raft_consensus.cc:729] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10ac6656c743d2b39587658c88728a, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.913569 8514 consensus_queue.cc:260] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.914006 8515 consensus_queue.cc:260] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.914132 8520 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:29.914320 8520 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:29.914317 8515 ts_tablet_manager.cc:1428] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:29.914647 8531 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:29.914814 8531 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:29.915418 8513 tablet_bootstrap.cc:492] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: No bootstrap required, opened a new log
I20260924 20:08:29.915511 8513 ts_tablet_manager.cc:1397] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 20:08:29.915939 8513 raft_consensus.cc:348] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.916110 8513 raft_consensus.cc:374] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:29.916152 8513 raft_consensus.cc:729] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7905d2f738d46ddbae7ca4d288386c3, State: Initialized, Role: FOLLOWER
I20260924 20:08:29.916288 8513 consensus_queue.cc:260] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:29.916558 8521 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:29.916692 8521 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:29.916949 8513 ts_tablet_manager.cc:1428] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 20:08:29.918522 8514 ts_tablet_manager.cc:1428] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260924 20:08:30.064271 8521 raft_consensus.cc:482] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:30.064440 8521 raft_consensus.cc:504] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.065759 8521 leader_election.cc:288] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:30.069968 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.066398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:37245 (local address 127.0.115.131:38673)
0924 20:08:30.066539 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.066543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.066579 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:08:30.066647 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.066650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.066690 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.066788 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:30.067080 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.067591 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.068860 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.069314 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.069441 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.069498 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.069747 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.069828 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:08:30.069968 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.066063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:59523 (local address 127.0.115.130:40167)
0924 20:08:30.066205 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.066212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.066471 (+ 259us) server_negotiation.cc:407] Connection header received
0924 20:08:30.066692 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.066697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.066753 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.066834 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:30.067119 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.067601 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.068854 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.069288 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.069413 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.069477 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.069752 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.069836 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:08:30.070000 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.066019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:38673 (local address 127.0.115.129:37245)
0924 20:08:30.066428 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.066442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.066566 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.066805 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.066808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.066828 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:30.066964 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.066969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.067717 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.067720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.068736 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.068746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.069434 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.069437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.069613 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.069644 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.069732 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.069838 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":84,"spinlock_wait_cycles":62592,"thread_start_us":97,"threads_started":1}
I20260924 20:08:30.070008 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.065967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:40167 (local address 127.0.115.129:59523)
0924 20:08:30.066380 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.066395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.066552 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.066859 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.066862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.066881 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:30.067008 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.067013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.067712 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.067718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.068730 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.068740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.069409 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.069412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.069604 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.069640 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.069730 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.069850 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":222,"spinlock_wait_cycles":5376,"thread_start_us":114,"threads_started":1}
I20260924 20:08:30.070555 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea2c24f9d544886902e8e870273bc60" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" is_pre_election: true
I20260924 20:08:30.070745 8439 raft_consensus.cc:2454] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 0.
I20260924 20:08:30.071038 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea2c24f9d544886902e8e870273bc60" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10ac6656c743d2b39587658c88728a" is_pre_election: true
I20260924 20:08:30.071168 8093 leader_election.cc:302] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c29aeb950ca4aa9ab4b55634fcb7473, b7905d2f738d46ddbae7ca4d288386c3; no voters:
I20260924 20:08:30.071270 8299 raft_consensus.cc:2454] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 0.
I20260924 20:08:30.071439 8521 raft_consensus.cc:2781] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:30.071506 8521 raft_consensus.cc:482] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:30.071548 8521 raft_consensus.cc:3037] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.072667 8521 raft_consensus.cc:504] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.073055 8521 leader_election.cc:288] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:30.073357 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea2c24f9d544886902e8e870273bc60" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10ac6656c743d2b39587658c88728a"
I20260924 20:08:30.073422 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bea2c24f9d544886902e8e870273bc60" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473"
I20260924 20:08:30.073493 8299 raft_consensus.cc:3037] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.073580 8439 raft_consensus.cc:3037] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.074442 8439 raft_consensus.cc:2454] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 1.
I20260924 20:08:30.074512 8299 raft_consensus.cc:2454] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 1.
I20260924 20:08:30.074751 8093 leader_election.cc:302] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c29aeb950ca4aa9ab4b55634fcb7473, b7905d2f738d46ddbae7ca4d288386c3; no voters:
I20260924 20:08:30.074958 8521 raft_consensus.cc:2781] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:30.075204 8521 raft_consensus.cc:686] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 LEADER]: Becoming Leader. State: Replica: b7905d2f738d46ddbae7ca4d288386c3, State: Running, Role: LEADER
I20260924 20:08:30.075507 8521 consensus_queue.cc:237] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.076769 8554 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: TxnStatusTablet state changed. Reason: New leader b7905d2f738d46ddbae7ca4d288386c3. Latest consensus state: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:30.076887 8554 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:08:30.077173 8556 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:08:30.077353 8556 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:08:30.078231 8002 catalog_manager.cc:5495] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129). New cstate: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:30.113051 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.109436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.115.129:43597)
0924 20:08:30.109560 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.109567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.109582 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:30.109823 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.109826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.109877 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.109983 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:30.110227 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.110742 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.111586 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.111811 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.111922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.112270 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.112655 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.112790 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.112867 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.112920 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:08:30.113054 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.109340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:43597 (local address 127.0.0.1:51422)
0924 20:08:30.109518 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.109534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.109649 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.109999 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.110002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.110010 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:30.110119 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.110124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.110856 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.110860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.111464 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.111472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.111921 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.111925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.112068 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.112112 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.112776 (+ 664us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.112791 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.112858 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.112940 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 20:08:30.116429 8439 raft_consensus.cc:1261] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Refusing update from remote peer b7905d2f738d46ddbae7ca4d288386c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.116567 8299 raft_consensus.cc:1261] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Refusing update from remote peer b7905d2f738d46ddbae7ca4d288386c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.116991 8521 consensus_queue.cc:1035] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.117175 8554 consensus_queue.cc:1035] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.119449 8520 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: TxnStatusTablet state changed. Reason: New leader b7905d2f738d46ddbae7ca4d288386c3. Latest consensus state: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:30.119575 8520 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:30.119544 8531 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: TxnStatusTablet state changed. Reason: New leader b7905d2f738d46ddbae7ca4d288386c3. Latest consensus state: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:30.119655 8531 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:30.121382 8554 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:30.121513 8554 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:08:30.121964 8531 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:30.122063 8531 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P 2d10ac6656c743d2b39587658c88728a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:30.122494 8526 mvcc.cc:205] Tried to move back new op lower bound from 7332988969431171072 to 7332988969272004608. Current Snapshot: MvccSnapshot[applied={T|T < 7332988969431171072 or (T in {7332988969431171072})}]
I20260924 20:08:30.125950 8521 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:30.126103 8521 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P b7905d2f738d46ddbae7ca4d288386c3: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:08:30.127308 8520 tablet_replica.cc:439] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } } }
I20260924 20:08:30.127427 8520 tablet_replica.cc:442] T bea2c24f9d544886902e8e870273bc60 P 6c29aeb950ca4aa9ab4b55634fcb7473: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:08:30.136701 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.132842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:43597 (local address 127.0.115.130:53341)
0924 20:08:30.133216 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.133230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.133315 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.133559 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.133562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.133568 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:30.133659 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.133663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.134759 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.134762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.135371 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.135379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.136035 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.136039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.136104 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.136148 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.136458 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.136462 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.136510 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.136576 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":101,"threads_started":1}
I20260924 20:08:30.136715 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.132938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:53341 (local address 127.0.115.129:43597)
0924 20:08:30.133054 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.133060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.133305 (+ 245us) server_negotiation.cc:407] Connection header received
0924 20:08:30.133392 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.133396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.133449 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.133537 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:30.133766 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.134642 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.135479 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.135862 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.135999 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.136279 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.136369 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.136450 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.136524 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.136580 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 20:08:30.143226 8159 raft_consensus.cc:1261] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Refusing update from remote peer 2d10ac6656c743d2b39587658c88728a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.143380 8439 raft_consensus.cc:1261] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Refusing update from remote peer 2d10ac6656c743d2b39587658c88728a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.143748 8531 consensus_queue.cc:1035] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.143898 8523 consensus_queue.cc:1035] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.148342 8528 mvcc.cc:205] Tried to move back new op lower bound from 7332988969543335936 to 7332988968371707904. Current Snapshot: MvccSnapshot[applied={T|T < 7332988969543335936}]
I20260924 20:08:30.154074 8527 mvcc.cc:205] Tried to move back new op lower bound from 7332988969543335936 to 7332988968371707904. Current Snapshot: MvccSnapshot[applied={T|T < 7332988969543335936}]
I20260924 20:08:30.169291 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.165024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:41581 (local address 127.0.115.130:40167)
0924 20:08:30.165129 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.165136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.165416 (+ 280us) server_negotiation.cc:407] Connection header received
0924 20:08:30.165497 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.165501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.165558 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.165648 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:30.165943 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.166845 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.168114 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.168375 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.168500 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.168842 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.168925 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.169004 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.169091 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.169143 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 20:08:30.169310 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.164956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:40167 (local address 127.0.115.129:41581)
0924 20:08:30.165316 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.165328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.165418 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.165649 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.165651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.165667 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:30.165833 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.165838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.166955 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.166958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.167858 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.167870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.168562 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.168566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.168652 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.168700 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.169011 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.169020 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.169083 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.169163 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260924 20:08:30.191289 8001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:30.191468 8001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:30.193529 8001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Sending DeleteTablet for 3 replicas of tablet bfc1a70c387340a6a120eb9d1e7ee15b
I20260924 20:08:30.194249 8279 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc1a70c387340a6a120eb9d1e7ee15b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:46874
I20260924 20:08:30.194447 8536 tablet_replica.cc:330] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: stopping tablet replica
I20260924 20:08:30.194602 8536 raft_consensus.cc:2229] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:30.194602 8419 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc1a70c387340a6a120eb9d1e7ee15b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260924 20:08:30.194833 8536 raft_consensus.cc:2258] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.194820 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc1a70c387340a6a120eb9d1e7ee15b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:51420
I20260924 20:08:30.195008 8538 tablet_replica.cc:330] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: stopping tablet replica
I20260924 20:08:30.195025 8537 tablet_replica.cc:330] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: stopping tablet replica
I20260924 20:08:30.195128 8537 raft_consensus.cc:2229] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:30.195127 8538 raft_consensus.cc:2229] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:30.195238 8538 raft_consensus.cc:2258] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.195238 8537 raft_consensus.cc:2258] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.196205 8536 ts_tablet_manager.cc:1905] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.196343 8538 ts_tablet_manager.cc:1905] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.199407 8538 ts_tablet_manager.cc:1918] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:30.199499 8538 log.cc:1196] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/bfc1a70c387340a6a120eb9d1e7ee15b
I20260924 20:08:30.199757 8536 ts_tablet_manager.cc:1918] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:30.199795 8538 ts_tablet_manager.cc:1939] T bfc1a70c387340a6a120eb9d1e7ee15b P b7905d2f738d46ddbae7ca4d288386c3: Deleting consensus metadata
I20260924 20:08:30.199844 8536 log.cc:1196] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/bfc1a70c387340a6a120eb9d1e7ee15b
I20260924 20:08:30.200119 8536 ts_tablet_manager.cc:1939] T bfc1a70c387340a6a120eb9d1e7ee15b P 2d10ac6656c743d2b39587658c88728a: Deleting consensus metadata
I20260924 20:08:30.200565 7986 catalog_manager.cc:4841] TS b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597): tablet bfc1a70c387340a6a120eb9d1e7ee15b (table test-table [id=68b7bc8c485144e29d18ef1a30be1f0c]) successfully deleted
I20260924 20:08:30.201175 7986 catalog_manager.cc:4841] TS 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167): tablet bfc1a70c387340a6a120eb9d1e7ee15b (table test-table [id=68b7bc8c485144e29d18ef1a30be1f0c]) successfully deleted
I20260924 20:08:30.201213 8537 ts_tablet_manager.cc:1905] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.204069 8537 ts_tablet_manager.cc:1918] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:30.204172 8537 log.cc:1196] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/bfc1a70c387340a6a120eb9d1e7ee15b
I20260924 20:08:30.204447 8537 ts_tablet_manager.cc:1939] T bfc1a70c387340a6a120eb9d1e7ee15b P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting consensus metadata
I20260924 20:08:30.205190 7989 catalog_manager.cc:4841] TS 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673): tablet bfc1a70c387340a6a120eb9d1e7ee15b (table test-table [id=68b7bc8c485144e29d18ef1a30be1f0c]) successfully deleted
Sep 24 20:08:30 dist-test-slave-2pm7 krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:08:30 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:30 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:08:30.223452 8051 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:30.215564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.0.115.190:44657)
0924 20:08:30.215726 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.215732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.215934 (+ 202us) server_negotiation.cc:407] Connection header received
0924 20:08:30.216127 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.216129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.216177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.216253 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:30.217268 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.217784 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.219153 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.219366 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.219442 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.223319 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53766: BlockingRecv error: recv got EOF from 127.0.0.1:53766 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 20:08:30 dist-test-slave-2pm7 krb5kdc[7957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM
I20260924 20:08:30.237879 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.225305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.0.115.190:44657)
0924 20:08:30.225423 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.225427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.228255 (+ 2828us) server_negotiation.cc:407] Connection header received
0924 20:08:30.228355 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.228358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.228417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.228515 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:30.229244 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.230147 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.230940 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.231314 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.231439 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.234208 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:30.234212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:30.234215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:30.234247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:30.236069 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:30.236500 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:30.236504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:30.236506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:30.236608 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:30.236884 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:30.236888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:30.236890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:30.237157 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:30.237312 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.237563 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.237695 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:08:30.240759 8001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:30.241274 8001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:30.246346 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 035c63243cd1402e8df965fd7f5cefe1 (DEFAULT_TABLE table=test-table [id=2f27213386f743d9a2485b23fe48a06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:30.246420 8279 tablet_service.cc:1467] Processing CreateTablet for tablet 035c63243cd1402e8df965fd7f5cefe1 (DEFAULT_TABLE table=test-table [id=2f27213386f743d9a2485b23fe48a06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:30.246713 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035c63243cd1402e8df965fd7f5cefe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:30.246708 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035c63243cd1402e8df965fd7f5cefe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:30.246793 8419 tablet_service.cc:1467] Processing CreateTablet for tablet 035c63243cd1402e8df965fd7f5cefe1 (DEFAULT_TABLE table=test-table [id=2f27213386f743d9a2485b23fe48a06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:30.247040 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035c63243cd1402e8df965fd7f5cefe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:30.249310 8515 tablet_bootstrap.cc:492] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: Bootstrap starting.
I20260924 20:08:30.249326 8514 tablet_bootstrap.cc:492] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: Bootstrap starting.
I20260924 20:08:30.249346 8513 tablet_bootstrap.cc:492] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: Bootstrap starting.
I20260924 20:08:30.250170 8514 tablet_bootstrap.cc:654] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:30.250170 8515 tablet_bootstrap.cc:654] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:30.250329 8513 tablet_bootstrap.cc:654] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:30.251657 8515 tablet_bootstrap.cc:492] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: No bootstrap required, opened a new log
I20260924 20:08:30.251744 8515 ts_tablet_manager.cc:1397] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:30.251842 8514 tablet_bootstrap.cc:492] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: No bootstrap required, opened a new log
I20260924 20:08:30.251907 8514 ts_tablet_manager.cc:1397] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 20:08:30.252210 8515 raft_consensus.cc:348] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.252339 8515 raft_consensus.cc:374] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:30.252377 8515 raft_consensus.cc:729] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10ac6656c743d2b39587658c88728a, State: Initialized, Role: FOLLOWER
I20260924 20:08:30.252343 8514 raft_consensus.cc:348] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.252441 8514 raft_consensus.cc:374] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:30.252472 8514 raft_consensus.cc:729] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c29aeb950ca4aa9ab4b55634fcb7473, State: Initialized, Role: FOLLOWER
I20260924 20:08:30.252504 8515 consensus_queue.cc:260] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.252576 8514 consensus_queue.cc:260] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.252770 8515 ts_tablet_manager.cc:1428] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:30.252828 8514 ts_tablet_manager.cc:1428] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:30.253612 8513 tablet_bootstrap.cc:492] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: No bootstrap required, opened a new log
I20260924 20:08:30.253702 8513 ts_tablet_manager.cc:1397] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260924 20:08:30.254192 8513 raft_consensus.cc:348] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.254302 8513 raft_consensus.cc:374] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:30.254334 8513 raft_consensus.cc:729] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7905d2f738d46ddbae7ca4d288386c3, State: Initialized, Role: FOLLOWER
I20260924 20:08:30.254457 8513 consensus_queue.cc:260] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.254701 8513 ts_tablet_manager.cc:1428] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:30.344306 8554 raft_consensus.cc:482] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:30.344445 8554 raft_consensus.cc:504] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.344936 8554 leader_election.cc:288] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:30.345391 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035c63243cd1402e8df965fd7f5cefe1" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10ac6656c743d2b39587658c88728a" is_pre_election: true
I20260924 20:08:30.345429 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035c63243cd1402e8df965fd7f5cefe1" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" is_pre_election: true
I20260924 20:08:30.345551 8299 raft_consensus.cc:2454] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 0.
I20260924 20:08:30.345589 8439 raft_consensus.cc:2454] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 0.
I20260924 20:08:30.345907 8093 leader_election.cc:302] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c29aeb950ca4aa9ab4b55634fcb7473, b7905d2f738d46ddbae7ca4d288386c3; no voters:
I20260924 20:08:30.346088 8554 raft_consensus.cc:2781] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:30.346167 8554 raft_consensus.cc:482] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:30.346215 8554 raft_consensus.cc:3037] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.347311 8554 raft_consensus.cc:504] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.347673 8554 leader_election.cc:288] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 election: Requested vote from peers 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167), 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673)
I20260924 20:08:30.348023 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035c63243cd1402e8df965fd7f5cefe1" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10ac6656c743d2b39587658c88728a"
I20260924 20:08:30.348052 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035c63243cd1402e8df965fd7f5cefe1" candidate_uuid: "b7905d2f738d46ddbae7ca4d288386c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473"
I20260924 20:08:30.348155 8439 raft_consensus.cc:3037] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.348163 8299 raft_consensus.cc:3037] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.349042 8299 raft_consensus.cc:2454] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 1.
I20260924 20:08:30.349359 8090 leader_election.cc:302] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10ac6656c743d2b39587658c88728a, b7905d2f738d46ddbae7ca4d288386c3; no voters:
I20260924 20:08:30.349547 8554 raft_consensus.cc:2781] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:30.349606 8439 raft_consensus.cc:2454] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7905d2f738d46ddbae7ca4d288386c3 in term 1.
I20260924 20:08:30.349655 8554 raft_consensus.cc:686] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 LEADER]: Becoming Leader. State: Replica: b7905d2f738d46ddbae7ca4d288386c3, State: Running, Role: LEADER
I20260924 20:08:30.349814 8554 consensus_queue.cc:237] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.351555 8002 catalog_manager.cc:5495] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129). New cstate: current_term: 1 leader_uuid: "b7905d2f738d46ddbae7ca4d288386c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } health_report { overall_health: UNKNOWN } } }
I20260924 20:08:30.382110 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.378945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.0.115.129:43597)
0924 20:08:30.379088 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.379101 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.379118 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:08:30.379174 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.379177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.379234 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.379333 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:30.379578 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.380160 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.381036 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.381257 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.381364 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.381621 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.381693 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.381773 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.381896 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.381950 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:08:30.384757 8299 raft_consensus.cc:1261] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Refusing update from remote peer b7905d2f738d46ddbae7ca4d288386c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.384733 8439 raft_consensus.cc:1261] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Refusing update from remote peer b7905d2f738d46ddbae7ca4d288386c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.385227 8521 consensus_queue.cc:1035] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.385381 8554 consensus_queue.cc:1035] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.395262 8002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:30.395432 8002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:30.397693 8002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Sending DeleteTablet for 3 replicas of tablet 035c63243cd1402e8df965fd7f5cefe1
I20260924 20:08:30.398299 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet 035c63243cd1402e8df965fd7f5cefe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:51420
I20260924 20:08:30.398422 8279 tablet_service.cc:1514] Processing DeleteTablet for tablet 035c63243cd1402e8df965fd7f5cefe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:46874
I20260924 20:08:30.398521 8538 tablet_replica.cc:330] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: stopping tablet replica
I20260924 20:08:30.398620 8536 tablet_replica.cc:330] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: stopping tablet replica
I20260924 20:08:30.398640 8538 raft_consensus.cc:2229] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:30.398757 8536 raft_consensus.cc:2229] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:30.398893 8538 raft_consensus.cc:2258] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.398888 8536 raft_consensus.cc:2258] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.399369 8419 tablet_service.cc:1514] Processing DeleteTablet for tablet 035c63243cd1402e8df965fd7f5cefe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260924 20:08:30.399561 8537 tablet_replica.cc:330] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: stopping tablet replica
I20260924 20:08:30.399657 8536 ts_tablet_manager.cc:1905] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.399691 8537 raft_consensus.cc:2229] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:30.399834 8537 raft_consensus.cc:2258] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.400228 8538 ts_tablet_manager.cc:1905] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.400583 8537 ts_tablet_manager.cc:1905] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.402179 8536 ts_tablet_manager.cc:1918] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:30.402253 8536 log.cc:1196] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/035c63243cd1402e8df965fd7f5cefe1
I20260924 20:08:30.402501 8536 ts_tablet_manager.cc:1939] T 035c63243cd1402e8df965fd7f5cefe1 P 2d10ac6656c743d2b39587658c88728a: Deleting consensus metadata
I20260924 20:08:30.402706 8538 ts_tablet_manager.cc:1918] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:30.402793 8538 log.cc:1196] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/035c63243cd1402e8df965fd7f5cefe1
I20260924 20:08:30.403003 8537 ts_tablet_manager.cc:1918] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:30.403054 8538 ts_tablet_manager.cc:1939] T 035c63243cd1402e8df965fd7f5cefe1 P b7905d2f738d46ddbae7ca4d288386c3: Deleting consensus metadata
I20260924 20:08:30.403084 8537 log.cc:1196] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/035c63243cd1402e8df965fd7f5cefe1
I20260924 20:08:30.403138 7986 catalog_manager.cc:4841] TS 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167): tablet 035c63243cd1402e8df965fd7f5cefe1 (table test-table [id=2f27213386f743d9a2485b23fe48a06e]) successfully deleted
I20260924 20:08:30.403319 8537 ts_tablet_manager.cc:1939] T 035c63243cd1402e8df965fd7f5cefe1 P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting consensus metadata
I20260924 20:08:30.404001 7989 catalog_manager.cc:4841] TS 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673): tablet 035c63243cd1402e8df965fd7f5cefe1 (table test-table [id=2f27213386f743d9a2485b23fe48a06e]) successfully deleted
I20260924 20:08:30.404515 7986 catalog_manager.cc:4841] TS b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597): tablet 035c63243cd1402e8df965fd7f5cefe1 (table test-table [id=2f27213386f743d9a2485b23fe48a06e]) successfully deleted
I20260924 20:08:30.404717 8002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:30.405174 8002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:30.409857 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 0026f38a84e04ad8b4b436ff85794054 (DEFAULT_TABLE table=test-table [id=c6c419f8683844ffa4d54bb2847371a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:30.409973 8279 tablet_service.cc:1467] Processing CreateTablet for tablet 0026f38a84e04ad8b4b436ff85794054 (DEFAULT_TABLE table=test-table [id=c6c419f8683844ffa4d54bb2847371a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:30.409989 8419 tablet_service.cc:1467] Processing CreateTablet for tablet 0026f38a84e04ad8b4b436ff85794054 (DEFAULT_TABLE table=test-table [id=c6c419f8683844ffa4d54bb2847371a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:30.410130 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026f38a84e04ad8b4b436ff85794054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:30.410233 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026f38a84e04ad8b4b436ff85794054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:30.410235 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026f38a84e04ad8b4b436ff85794054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:30.412456 8514 tablet_bootstrap.cc:492] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: Bootstrap starting.
I20260924 20:08:30.412479 8513 tablet_bootstrap.cc:492] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3: Bootstrap starting.
I20260924 20:08:30.412680 8515 tablet_bootstrap.cc:492] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: Bootstrap starting.
I20260924 20:08:30.413311 8513 tablet_bootstrap.cc:654] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:30.413348 8514 tablet_bootstrap.cc:654] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:30.413448 8515 tablet_bootstrap.cc:654] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:30.415022 8513 tablet_bootstrap.cc:492] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3: No bootstrap required, opened a new log
I20260924 20:08:30.415114 8513 ts_tablet_manager.cc:1397] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 20:08:30.415374 8515 tablet_bootstrap.cc:492] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: No bootstrap required, opened a new log
I20260924 20:08:30.415460 8515 ts_tablet_manager.cc:1397] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 20:08:30.415561 8513 raft_consensus.cc:348] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.415697 8513 raft_consensus.cc:374] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:30.415736 8513 raft_consensus.cc:729] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7905d2f738d46ddbae7ca4d288386c3, State: Initialized, Role: FOLLOWER
I20260924 20:08:30.415827 8515 raft_consensus.cc:348] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.415860 8513 consensus_queue.cc:260] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.415961 8515 raft_consensus.cc:374] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:30.415997 8515 raft_consensus.cc:729] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10ac6656c743d2b39587658c88728a, State: Initialized, Role: FOLLOWER
I20260924 20:08:30.416112 8515 consensus_queue.cc:260] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.416173 8513 ts_tablet_manager.cc:1428] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:30.416379 8515 ts_tablet_manager.cc:1428] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:30.417239 8514 tablet_bootstrap.cc:492] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: No bootstrap required, opened a new log
I20260924 20:08:30.417346 8514 ts_tablet_manager.cc:1397] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 20:08:30.417831 8514 raft_consensus.cc:348] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.417953 8514 raft_consensus.cc:374] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:30.417994 8514 raft_consensus.cc:729] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c29aeb950ca4aa9ab4b55634fcb7473, State: Initialized, Role: FOLLOWER
I20260924 20:08:30.418125 8514 consensus_queue.cc:260] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.418406 8514 ts_tablet_manager.cc:1428] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:08:30.495764 8520 raft_consensus.cc:482] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:30.495900 8520 raft_consensus.cc:504] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.497056 8520 leader_election.cc:288] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597), 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167)
I20260924 20:08:30.502068 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.497847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:53225 (local address 127.0.115.130:40167)
0924 20:08:30.497976 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.497983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.498241 (+ 258us) server_negotiation.cc:407] Connection header received
0924 20:08:30.498404 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.498407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.498455 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.498535 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:30.498883 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.499604 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.500792 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.501377 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.501498 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.501555 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.501841 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.501943 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:08:30.502077 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.497732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:40167 (local address 127.0.115.131:53225)
0924 20:08:30.498128 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.498154 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.498304 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.498574 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.498577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.498597 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:30.498747 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.498754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.499739 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.499743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.500663 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.500672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.501541 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.501545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.501659 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.501686 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.501816 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.501897 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":19,"thread_start_us":75,"threads_started":1}
I20260924 20:08:30.502153 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.497439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:51051 (local address 127.0.115.129:43597)
0924 20:08:30.497578 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.497583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.497768 (+ 185us) server_negotiation.cc:407] Connection header received
0924 20:08:30.498023 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.498027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.498124 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.498185 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:30.498645 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.499393 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.500970 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.501557 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.501680 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.501732 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.501958 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.502032 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:08:30.502590 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026f38a84e04ad8b4b436ff85794054" candidate_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10ac6656c743d2b39587658c88728a" is_pre_election: true
I20260924 20:08:30.502769 8299 raft_consensus.cc:2454] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c29aeb950ca4aa9ab4b55634fcb7473 in term 0.
I20260924 20:08:30.502948 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.497363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:43597 (local address 127.0.115.131:51051)
0924 20:08:30.497651 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.497689 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.497877 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.498364 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.498366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.498387 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:30.498520 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.498529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.499489 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.499492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.500835 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.500847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.501687 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.501690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.501819 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.501842 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.502649 (+ 807us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.502737 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":22,"thread_start_us":81,"threads_started":1}
I20260924 20:08:30.503170 8370 leader_election.cc:302] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10ac6656c743d2b39587658c88728a, 6c29aeb950ca4aa9ab4b55634fcb7473; no voters:
I20260924 20:08:30.503473 8520 raft_consensus.cc:2781] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:30.503545 8520 raft_consensus.cc:482] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:30.503587 8520 raft_consensus.cc:3037] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.503895 8159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026f38a84e04ad8b4b436ff85794054" candidate_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7905d2f738d46ddbae7ca4d288386c3" is_pre_election: true
I20260924 20:08:30.504079 8159 raft_consensus.cc:2454] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c29aeb950ca4aa9ab4b55634fcb7473 in term 0.
I20260924 20:08:30.504801 8520 raft_consensus.cc:504] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.505170 8520 leader_election.cc:288] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [CANDIDATE]: Term 1 election: Requested vote from peers b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597), 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167)
I20260924 20:08:30.505577 8159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026f38a84e04ad8b4b436ff85794054" candidate_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7905d2f738d46ddbae7ca4d288386c3"
I20260924 20:08:30.505615 8299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026f38a84e04ad8b4b436ff85794054" candidate_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10ac6656c743d2b39587658c88728a"
I20260924 20:08:30.505713 8159 raft_consensus.cc:3037] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.505757 8299 raft_consensus.cc:3037] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:30.506655 8299 raft_consensus.cc:2454] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c29aeb950ca4aa9ab4b55634fcb7473 in term 1.
I20260924 20:08:30.506865 8159 raft_consensus.cc:2454] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c29aeb950ca4aa9ab4b55634fcb7473 in term 1.
I20260924 20:08:30.507002 8370 leader_election.cc:302] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10ac6656c743d2b39587658c88728a, 6c29aeb950ca4aa9ab4b55634fcb7473; no voters:
I20260924 20:08:30.507227 8520 raft_consensus.cc:2781] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:30.507437 8520 raft_consensus.cc:686] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 LEADER]: Becoming Leader. State: Replica: 6c29aeb950ca4aa9ab4b55634fcb7473, State: Running, Role: LEADER
I20260924 20:08:30.507714 8520 consensus_queue.cc:237] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } }
I20260924 20:08:30.510535 8001 catalog_manager.cc:5495] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131). New cstate: current_term: 1 leader_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c29aeb950ca4aa9ab4b55634fcb7473" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38673 } health_report { overall_health: HEALTHY } } }
I20260924 20:08:30.527287 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.523522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.0.115.131:38673)
0924 20:08:30.523637 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.523642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.523670 (+ 28us) server_negotiation.cc:407] Connection header received
0924 20:08:30.523886 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.523892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.523987 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.524128 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:30.524414 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.524980 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.525820 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.526041 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.526151 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.526443 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.526896 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.527031 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.527092 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.527145 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 20:08:30.538360 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.534437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:43597 (local address 127.0.115.131:50693)
0924 20:08:30.534746 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.534777 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.534902 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.535192 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.535194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.535202 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:30.535306 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.535310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.536337 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.536340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.536947 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.536955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.537547 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.537550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.537680 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.537727 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.538093 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.538099 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.538150 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.538217 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
I20260924 20:08:30.538362 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.534576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:50693 (local address 127.0.115.129:43597)
0924 20:08:30.534720 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.534725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.534859 (+ 134us) server_negotiation.cc:407] Connection header received
0924 20:08:30.535011 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.535014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.535069 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.535162 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:30.535409 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.536224 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.537064 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.537429 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.537548 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.537894 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.537998 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.538085 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.538163 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.538218 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:08:30.545537 8299 raft_consensus.cc:1261] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Refusing update from remote peer 6c29aeb950ca4aa9ab4b55634fcb7473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.545571 8159 raft_consensus.cc:1261] T 0026f38a84e04ad8b4b436ff85794054 P b7905d2f738d46ddbae7ca4d288386c3 [term 1 FOLLOWER]: Refusing update from remote peer 6c29aeb950ca4aa9ab4b55634fcb7473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:30.546046 8520 consensus_queue.cc:1035] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d10ac6656c743d2b39587658c88728a" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.546268 8595 consensus_queue.cc:1035] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7905d2f738d46ddbae7ca4d288386c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:30.564337 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.561111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:47769 (local address 127.0.115.131:38673)
0924 20:08:30.561235 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:30.561240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:30.561295 (+ 55us) server_negotiation.cc:407] Connection header received
0924 20:08:30.561414 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:30.561416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:30.561454 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:30.561545 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:30.561757 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.562352 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.563188 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.563492 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.563603 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.563859 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.563978 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.564045 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:30.564148 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 20:08:30.564198 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:08:30.564365 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.561072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:38673 (local address 127.0.115.129:47769)
0924 20:08:30.561248 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.561260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:30.561332 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:30.561555 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:30.561559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:30.561567 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:08:30.561665 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.561671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.562462 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.562467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.563052 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:30.563060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:30.563615 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:30.563619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:30.563715 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:30.563740 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:08:30.564049 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:08:30.564058 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:08:30.564136 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 20:08:30.564226 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 20:08:30.587409 8002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:30.587580 8002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:30.589761 8002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 807de03cc1d547cabeb1efc45c05e932: Sending DeleteTablet for 3 replicas of tablet 0026f38a84e04ad8b4b436ff85794054
I20260924 20:08:30.590441 8419 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026f38a84e04ad8b4b436ff85794054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260924 20:08:30.590632 8537 tablet_replica.cc:330] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: stopping tablet replica
I20260924 20:08:30.590600 8279 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026f38a84e04ad8b4b436ff85794054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:30 UTC) from {username='oryx', principal='oryx/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:46874
I20260924 20:08:30.590745 8537 raft_consensus.cc:2229] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:30.590776 8536 tablet_replica.cc:330] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: stopping tablet replica
I20260924 20:08:30.591028 8537 raft_consensus.cc:2258] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.591099 8536 raft_consensus.cc:2229] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:30.591221 8536 raft_consensus.cc:2258] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:30.592020 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 8074
I20260924 20:08:30.592033 8536 ts_tablet_manager.cc:1905] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.594924 8536 ts_tablet_manager.cc:1918] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:30.595026 8536 log.cc:1196] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/0026f38a84e04ad8b4b436ff85794054
I20260924 20:08:30.595261 8536 ts_tablet_manager.cc:1939] T 0026f38a84e04ad8b4b436ff85794054 P 2d10ac6656c743d2b39587658c88728a: Deleting consensus metadata
I20260924 20:08:30.596240 7986 catalog_manager.cc:4841] TS 2d10ac6656c743d2b39587658c88728a (127.0.115.130:40167): tablet 0026f38a84e04ad8b4b436ff85794054 (table test-table [id=c6c419f8683844ffa4d54bb2847371a8]) successfully deleted
I20260924 20:08:30.597496 8537 ts_tablet_manager.cc:1905] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:30.600606 8537 ts_tablet_manager.cc:1918] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:08:30.600682 8537 log.cc:1196] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/0026f38a84e04ad8b4b436ff85794054
I20260924 20:08:30.600929 8537 ts_tablet_manager.cc:1939] T 0026f38a84e04ad8b4b436ff85794054 P 6c29aeb950ca4aa9ab4b55634fcb7473: Deleting consensus metadata
I20260924 20:08:30.601768 7989 catalog_manager.cc:4841] TS 6c29aeb950ca4aa9ab4b55634fcb7473 (127.0.115.131:38673): tablet 0026f38a84e04ad8b4b436ff85794054 (table test-table [id=c6c419f8683844ffa4d54bb2847371a8]) successfully deleted
W20260924 20:08:30.604914 7986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:43597 (error 108)
I20260924 20:08:30.605485 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 8214
I20260924 20:08:30.608285 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:30.607861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:43597 (local address 127.0.0.1:40996)
0924 20:08:30.608088 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:30.608167 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:43597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":201,"threads_started":1}
W20260924 20:08:30.608464 7986 catalog_manager.cc:4584] TS b7905d2f738d46ddbae7ca4d288386c3 (127.0.115.129:43597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0026f38a84e04ad8b4b436ff85794054: Network error: Client connection negotiation failed: client connection to 127.0.115.129:43597: connect: Connection refused (error 111)
I20260924 20:08:30.614500 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 8354
I20260924 20:08:30.620790 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 7970
2026-09-24T20:08:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3205 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:30 dist-test-slave-2pm7 krb5kdc[8608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:30 dist-test-slave-2pm7 krb5kdc[8608](info): set up 2 sockets
Sep 24 20:08:30 dist-test-slave-2pm7 krb5kdc[8608](info): commencing operation
krb5kdc: starting...
W20260924 20:08:31.664884 462 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:31 dist-test-slave-2pm7 krb5kdc[8608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:31.790697 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39237
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:39237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:31.903424 8621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:31.903707 8621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:31.903746 8621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:31.907732 8621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:31.907788 8621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:31.907809 8621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:31.907824 8621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:31.907893 8621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:31.913271 8621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:39237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:31.914417 8621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:31.915490 8621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:31.924155 8626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:31.924314 8629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:31.924257 8627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:31.924229 8621 server_base.cc:1034] running on GCE node
I20260924 20:08:31.924932 8621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:31.926093 8621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:31.927335 8621 hybrid_clock.cc:648] HybridClock initialized: now 1790280511927294 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T20:08:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1316 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:31 dist-test-slave-2pm7 krb5kdc[8634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:31 dist-test-slave-2pm7 krb5kdc[8634](info): set up 2 sockets
Sep 24 20:08:31 dist-test-slave-2pm7 krb5kdc[8634](info): commencing operation
krb5kdc: starting...
W20260924 20:08:33.016891 462 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:33 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:33.142998 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:38589
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:33.254539 8647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:33.254855 8647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:33.254923 8647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:33.258919 8647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:33.259011 8647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:33.259050 8647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:33.259088 8647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:33.259122 8647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:33.264562 8647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:33.265648 8647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:33.266681 8647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:33.275476 8653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:33.275485 8652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:33.275561 8647 server_base.cc:1034] running on GCE node
W20260924 20:08:33.275488 8655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:33.276401 8647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:33.277490 8647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:33.278656 8647 hybrid_clock.cc:648] HybridClock initialized: now 1790280513278636 us; error 28 us; skew 500 ppm
Sep 24 20:08:33 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:33.286317 8647 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:33.287281 8647 webserver.cc:460] Webserver started at http://127.0.115.190:40563/ using document root <none> and password file <none>
I20260924 20:08:33.287869 8647 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:33.287981 8647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:33.288225 8647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:33.290388 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "8d7df1e5d4694f93bea4494dbf7ebe51"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "c7e8bdffb1c37a3fc73a76039ef58a92"
server_key_iv: "c22a3cfd142afda6fc122b406cb564f6"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.290971 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "8d7df1e5d4694f93bea4494dbf7ebe51"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "c7e8bdffb1c37a3fc73a76039ef58a92"
server_key_iv: "c22a3cfd142afda6fc122b406cb564f6"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.294770 8647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 20:08:33.297397 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:33.298816 8647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:08:33.298970 8647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "8d7df1e5d4694f93bea4494dbf7ebe51"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "c7e8bdffb1c37a3fc73a76039ef58a92"
server_key_iv: "c22a3cfd142afda6fc122b406cb564f6"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.299088 8647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:33.318401 8647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:33.321478 8647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:33.321695 8647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:33.329214 8647 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:38589
I20260924 20:08:33.329227 8714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:38589 every 8 connection(s)
I20260924 20:08:33.330216 8647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:33.333002 8715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:33.338781 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Bootstrap starting.
I20260924 20:08:33.339571 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 8647
I20260924 20:08:33.339712 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:33.340044 462 external_mini_cluster.cc:1428] Setting key edc297d59be95015ed105c29b4dfa0b8
I20260924 20:08:33.341926 8715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:33.342689 8715 log.cc:824] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:33.345023 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: No bootstrap required, opened a new log
Sep 24 20:08:33 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:33.349754 8715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:33.349988 8715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:33.350047 8715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7df1e5d4694f93bea4494dbf7ebe51, State: Initialized, Role: FOLLOWER
I20260924 20:08:33.350548 8715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:33.350736 8715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:33.350832 8715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:33.350948 8715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:33.352217 8715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:33.352664 8715 leader_election.cc:302] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d7df1e5d4694f93bea4494dbf7ebe51; no voters:
I20260924 20:08:33.353026 8715 leader_election.cc:288] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:33.353137 8720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:33.353431 8720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7df1e5d4694f93bea4494dbf7ebe51, State: Running, Role: LEADER
I20260924 20:08:33.353869 8720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:33.354494 8715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:33.355417 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d7df1e5d4694f93bea4494dbf7ebe51. Latest consensus state: current_term: 1 leader_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } } }
I20260924 20:08:33.356125 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } } }
I20260924 20:08:33.356244 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:33.356328 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:33.357558 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:33.341387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.0.115.190:38589)
0924 20:08:33.341839 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:33.341854 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:33.341890 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:08:33.342620 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:33.342640 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:33.343080 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:33.343480 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:33.344366 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.345476 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.346669 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.347162 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.347290 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:33.350207 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:33.350213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:33.350228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:33.350257 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:33.352357 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:33.352987 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:33.352994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:33.353000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:33.353209 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:33.353824 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:33.353827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:33.353829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:33.354367 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:33.354545 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:33.355037 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 20:08:33.355381 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":128,"threads_started":1}
W20260924 20:08:33.359647 8735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:08:33.359727 8735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:08:33.360028 8732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:33.360767 8732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:33.366856 8732 catalog_manager.cc:1347] Generated new cluster ID: 6d62583539aa42949be0aefb3e609867
I20260924 20:08:33.366925 8732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:33.373615 8732 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:33.374583 8732 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:33.383267 8732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Generated new TSK 0
I20260924 20:08:33.384218 8732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:08:33.416515 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:38589
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:33.535279 8741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:33.535576 8741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:33.535629 8741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:33.539664 8741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:33.539731 8741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:33.539791 8741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:33.545334 8741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:33.546500 8741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:33.547524 8741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:33.557667 8747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:33.557797 8741 server_base.cc:1034] running on GCE node
W20260924 20:08:33.557718 8749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:33.557744 8746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:33.558316 8741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:33.559003 8741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:33.560181 8741 hybrid_clock.cc:648] HybridClock initialized: now 1790280513560148 us; error 46 us; skew 500 ppm
Sep 24 20:08:33 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:33.562855 8741 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:08:33.563706 8741 webserver.cc:460] Webserver started at http://127.0.115.129:37209/ using document root <none> and password file <none>
I20260924 20:08:33.564322 8741 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:33.564410 8741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:33.564648 8741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:33.566681 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "9e697b81c7554a7798d0947e8a3b5afd"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "d33d6322a3bed4d88392f82abdb30c84"
server_key_iv: "a0771e77e09bafba5f783a8b28179aa7"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.567279 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "9e697b81c7554a7798d0947e8a3b5afd"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "d33d6322a3bed4d88392f82abdb30c84"
server_key_iv: "a0771e77e09bafba5f783a8b28179aa7"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.571065 8741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:08:33.573719 8756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:33.574952 8741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:33.575080 8741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "9e697b81c7554a7798d0947e8a3b5afd"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "d33d6322a3bed4d88392f82abdb30c84"
server_key_iv: "a0771e77e09bafba5f783a8b28179aa7"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.575193 8741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:33.589671 8741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:33.593066 8741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:33.593350 8741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:33.594190 8741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:33.595435 8741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:33.595505 8741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:33.595562 8741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:33.595594 8741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:33.605118 8741 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:40635
I20260924 20:08:33.605221 8869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:40635 every 8 connection(s)
I20260924 20:08:33.606105 8741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:33.613471 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 8741
I20260924 20:08:33.613619 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:33.613936 462 external_mini_cluster.cc:1428] Setting key f91749088994fef2a9b8d200979926ae
Sep 24 20:08:33 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:33.621598 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:33.608205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:46653 (local address 127.0.115.190:38589)
0924 20:08:33.608356 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:33.608361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:33.609380 (+ 1019us) server_negotiation.cc:407] Connection header received
0924 20:08:33.610782 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:33.610785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:33.610834 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:33.610912 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:33.612150 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.612681 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.614246 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.614571 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.614655 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:33.617757 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:33.617760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:33.617762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:33.617787 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:33.619297 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:33.619934 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:33.619990 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:33.619993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:33.620090 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:33.620449 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:33.620454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:33.620456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:33.620655 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:33.620767 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:33.621293 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 20:08:33.621428 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:08:33.622491 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:33.608706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:46653)
0924 20:08:33.609206 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:33.609259 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:33.610570 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:33.611081 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:33.611089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:33.611462 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:33.612017 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:33.612032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.612808 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.612811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:33.614127 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:33.614140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.614729 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.614735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:33.614901 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:33.615799 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:33.615816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:33.617534 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:33.619436 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:33.619442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:33.619465 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:33.619820 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:33.620197 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:33.620201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:33.620203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:33.620348 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:33.620766 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:33.620774 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:33.621046 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 20:08:33.621251 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 20:08:33.621627 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:08:33.624012 8870 heartbeater.cc:347] Connected to a master server at 127.0.115.190:38589
I20260924 20:08:33.624274 8870 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:33.624845 8870 heartbeater.cc:511] Master 127.0.115.190:38589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:08:33.626533 8679 ts_manager.cc:194] Registered new tserver with Master: 9e697b81c7554a7798d0947e8a3b5afd (127.0.115.129:40635)
I20260924 20:08:33.627674 8679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:46653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:08:33.643221 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:38589
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:33.757283 8875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:33.757543 8875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:33.757586 8875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:33.761783 8875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:33.761848 8875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:33.761940 8875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:33.767390 8875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:33.768545 8875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:33.769595 8875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:33.779731 8881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:33.779685 8883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:33.779750 8880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:33.779842 8875 server_base.cc:1034] running on GCE node
I20260924 20:08:33.780505 8875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:33.781179 8875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:33.782394 8875 hybrid_clock.cc:648] HybridClock initialized: now 1790280513782356 us; error 51 us; skew 500 ppm
Sep 24 20:08:33 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:33.785394 8875 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:08:33.786439 8875 webserver.cc:460] Webserver started at http://127.0.115.130:33989/ using document root <none> and password file <none>
I20260924 20:08:33.787062 8875 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:33.787160 8875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:33.787400 8875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:33.789665 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "db1da0bfc7794144a6308fc5cf4919d4"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "a012a262c0b17b0526df049aa1edc98a"
server_key_iv: "7d837bea864020318367e9055b24a760"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.790273 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "db1da0bfc7794144a6308fc5cf4919d4"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "a012a262c0b17b0526df049aa1edc98a"
server_key_iv: "7d837bea864020318367e9055b24a760"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.794245 8875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:33.797037 8890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:33.798385 8875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:33.798542 8875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "db1da0bfc7794144a6308fc5cf4919d4"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "a012a262c0b17b0526df049aa1edc98a"
server_key_iv: "7d837bea864020318367e9055b24a760"
server_key_version: "encryptionkey@0"
I20260924 20:08:33.798689 8875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:33.817564 8875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:33.821077 8875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:33.821318 8875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:33.822044 8875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:33.823015 8875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:33.823117 8875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:33.823202 8875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:33.823259 8875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:33.833673 8875 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:36939
I20260924 20:08:33.833693 9003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:36939 every 8 connection(s)
I20260924 20:08:33.834753 8875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:33.840269 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 8875
I20260924 20:08:33.840430 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:33.840765 462 external_mini_cluster.cc:1428] Setting key 8a388848ea9b512f0cf52eb08bc7e3a0
Sep 24 20:08:33 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:33.849720 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:33.836885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:37689 (local address 127.0.115.190:38589)
0924 20:08:33.837041 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:33.837045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:33.837943 (+ 898us) server_negotiation.cc:407] Connection header received
0924 20:08:33.839298 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:33.839311 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:33.839361 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:33.839439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:33.840808 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.841512 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.842954 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.843349 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.843472 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:33.846044 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:33.846052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:33.846059 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:33.846092 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:33.847580 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:33.848155 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:33.848159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:33.848161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:33.848246 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:33.848598 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:33.848602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:33.848604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:33.848804 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:33.848915 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:33.849415 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 20:08:33.849534 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:08:33.850538 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:33.837318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:37689)
0924 20:08:33.837786 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:33.837830 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:33.839073 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:33.839612 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:33.839619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:33.840034 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:33.840649 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:33.840666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.841654 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.841658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:33.842822 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:33.842832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:33.843499 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:33.843502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:33.843734 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:33.844413 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:33.844429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:33.845874 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:33.847705 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:33.847714 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:33.847730 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:33.848051 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:33.848350 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:33.848353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:33.848355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:33.848495 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:33.848912 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:33.848920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:33.849164 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 20:08:33.849356 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 20:08:33.849684 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:08:33.851719 9004 heartbeater.cc:347] Connected to a master server at 127.0.115.190:38589
I20260924 20:08:33.852018 9004 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:33.852525 9004 heartbeater.cc:511] Master 127.0.115.190:38589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:08:33.853663 8679 ts_manager.cc:194] Registered new tserver with Master: db1da0bfc7794144a6308fc5cf4919d4 (127.0.115.130:36939)
I20260924 20:08:33.854246 8679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:37689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:08:33.872025 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:38589
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:33.993060 9009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:33.993335 9009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:33.993379 9009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:33.997638 9009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:33.997717 9009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:33.997779 9009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:34.003520 9009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:34.004710 9009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:34.005822 9009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:34.016144 9015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.016147 9014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.016201 9017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:34.016216 9009 server_base.cc:1034] running on GCE node
I20260924 20:08:34.016930 9009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:34.017691 9009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:34.018911 9009 hybrid_clock.cc:648] HybridClock initialized: now 1790280514018872 us; error 52 us; skew 500 ppm
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:34.022061 9009 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:34.022949 9009 webserver.cc:460] Webserver started at http://127.0.115.131:37903/ using document root <none> and password file <none>
I20260924 20:08:34.023566 9009 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:34.023658 9009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:34.023907 9009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:34.026062 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "ab5a4205cbd4433d973fe3b11d1b9323"
format_stamp: "Formatted at 2026-09-24 20:08:34 on dist-test-slave-2pm7"
server_key: "1928b6dffca27d1d4939a23873597e58"
server_key_iv: "7b167bfebcf8b3192f70e2a946130dda"
server_key_version: "encryptionkey@0"
I20260924 20:08:34.026638 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "ab5a4205cbd4433d973fe3b11d1b9323"
format_stamp: "Formatted at 2026-09-24 20:08:34 on dist-test-slave-2pm7"
server_key: "1928b6dffca27d1d4939a23873597e58"
server_key_iv: "7b167bfebcf8b3192f70e2a946130dda"
server_key_version: "encryptionkey@0"
I20260924 20:08:34.030457 9009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 20:08:34.033183 9024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.034463 9009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:08:34.034607 9009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "ab5a4205cbd4433d973fe3b11d1b9323"
format_stamp: "Formatted at 2026-09-24 20:08:34 on dist-test-slave-2pm7"
server_key: "1928b6dffca27d1d4939a23873597e58"
server_key_iv: "7b167bfebcf8b3192f70e2a946130dda"
server_key_version: "encryptionkey@0"
I20260924 20:08:34.034737 9009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:34.051731 9009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:34.055613 9009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:34.055836 9009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:34.056610 9009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:34.057497 9009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:34.057569 9009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.057644 9009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:34.057688 9009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.067732 9009 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:34113
I20260924 20:08:34.067775 9137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:34113 every 8 connection(s)
I20260924 20:08:34.068750 9009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:34.079149 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9009
I20260924 20:08:34.079303 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:34.079648 462 external_mini_cluster.cc:1428] Setting key 33029cf5d68857376313881259735472
I20260924 20:08:34.083738 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.070723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:43929 (local address 127.0.115.190:38589)
0924 20:08:34.070865 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.070868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.071772 (+ 904us) server_negotiation.cc:407] Connection header received
0924 20:08:34.073265 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.073269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.073331 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.073410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.074658 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.075211 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.076467 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.076701 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.076815 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.079647 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:34.079651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:34.079654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:34.079689 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:34.081498 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:34.082151 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:34.082155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:34.082157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:34.082240 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:34.082596 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:34.082600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:34.082601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:34.082799 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:34.082912 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:34.083444 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 20:08:34.083566 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:08:34.084681 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.071117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:43929)
0924 20:08:34.071590 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.071637 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.073034 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.073603 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.073610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.073980 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.074506 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.074520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.075346 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.075354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.076332 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.076342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.076865 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.076869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.077043 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.077673 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.077690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.079467 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:34.081653 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:34.081659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:34.081677 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:34.082024 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:34.082338 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:34.082341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:34.082343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:34.082492 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:34.082911 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:34.082919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:34.083202 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 20:08:34.083404 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 20:08:34.083753 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260924 20:08:34.086000 9138 heartbeater.cc:347] Connected to a master server at 127.0.115.190:38589
I20260924 20:08:34.086267 9138 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:34.086747 9138 heartbeater.cc:511] Master 127.0.115.190:38589 requested a full tablet report, sending...
I20260924 20:08:34.087770 8679 ts_manager.cc:194] Registered new tserver with Master: ab5a4205cbd4433d973fe3b11d1b9323 (127.0.115.131:34113)
I20260924 20:08:34.088491 8679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:43929
I20260924 20:08:34.093709 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab.
Entry for principal oryx/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab.
WARNING: no policy specified for oryx/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab.
Entry for principal oryx/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab.
WARNING: no policy specified for oryx/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab.
Entry for principal oryx/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab.
I20260924 20:08:34.171049 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 8741
I20260924 20:08:34.178182 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 8875
I20260924 20:08:34.184242 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9009
I20260924 20:08:34.189726 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 8647
I20260924 20:08:34.196326 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:38589
--webserver_interface=127.0.115.190
--webserver_port=40563
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:34.307283 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:34.307641 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:34.307729 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:34.312121 9147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:34.312247 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:34.312287 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:34.312326 9147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:34.312363 9147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:34.317896 9147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:34.319103 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:34.320216 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:34.328860 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.328850 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:34.329169 9147 server_base.cc:1034] running on GCE node
W20260924 20:08:34.328850 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:34.329967 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:34.331108 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:34.332268 9147 hybrid_clock.cc:648] HybridClock initialized: now 1790280514332242 us; error 39 us; skew 500 ppm
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:34.335119 9147 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:34.336089 9147 webserver.cc:460] Webserver started at http://127.0.115.190:40563/ using document root <none> and password file <none>
I20260924 20:08:34.336671 9147 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:34.336750 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:34.340809 9147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 20:08:34.342998 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.344372 9147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:34.344511 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "8d7df1e5d4694f93bea4494dbf7ebe51"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "c7e8bdffb1c37a3fc73a76039ef58a92"
server_key_iv: "c22a3cfd142afda6fc122b406cb564f6"
server_key_version: "encryptionkey@0"
I20260924 20:08:34.345072 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:34.368525 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:34.371559 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:34.371779 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:34.379182 9147 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:38589
I20260924 20:08:34.379212 9214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:38589 every 8 connection(s)
I20260924 20:08:34.380321 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:34.382411 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9147
I20260924 20:08:34.382897 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:40635
--local_ip_for_outbound_sockets=127.0.115.129
--tserver_master_addrs=127.0.115.190:38589
--webserver_port=37209
--webserver_interface=127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:08:34.385372 9215 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:08:34.387161 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Bootstrap starting.
I20260924 20:08:34.404726 9215 log.cc:824] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:34.411298 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:08:34.411755 9215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Bootstrap complete.
I20260924 20:08:34.416152 9215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:34.416558 9215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7df1e5d4694f93bea4494dbf7ebe51, State: Initialized, Role: FOLLOWER
I20260924 20:08:34.417096 9215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:34.417281 9215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:34.417368 9215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:34.417464 9215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:08:34.419413 9215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:34.419754 9215 leader_election.cc:302] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d7df1e5d4694f93bea4494dbf7ebe51; no voters:
I20260924 20:08:34.420116 9215 leader_election.cc:288] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:08:34.420243 9219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:08:34.420452 9219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [term 2 LEADER]: Becoming Leader. State: Replica: 8d7df1e5d4694f93bea4494dbf7ebe51, State: Running, Role: LEADER
I20260924 20:08:34.420751 9219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } }
I20260924 20:08:34.421393 9215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:34.422160 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } } }
I20260924 20:08:34.422289 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:34.422587 9221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d7df1e5d4694f93bea4494dbf7ebe51. Latest consensus state: current_term: 2 leader_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d7df1e5d4694f93bea4494dbf7ebe51" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 38589 } } }
I20260924 20:08:34.422732 9221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51 [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:34.423470 9228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:34.426563 9228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:34.427204 9228 catalog_manager.cc:1259] Loaded cluster ID: 6d62583539aa42949be0aefb3e609867
I20260924 20:08:34.427263 9228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:34.429751 9228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:34.430429 9228 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d7df1e5d4694f93bea4494dbf7ebe51: Loaded TSK: 0
I20260924 20:08:34.431646 9228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:08:34.500947 9217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:34.501269 9217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:34.501362 9217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:34.505522 9217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:34.505609 9217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:34.505704 9217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:34.510990 9217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.129.keytab
--principal=oryx/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:40635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=37209
--tserver_master_addrs=127.0.115.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:34.512099 9217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:34.513159 9217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:34.523128 9242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.523260 9244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.523157 9241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:34.523478 9217 server_base.cc:1034] running on GCE node
I20260924 20:08:34.524081 9217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:34.524783 9217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:34.525974 9217 hybrid_clock.cc:648] HybridClock initialized: now 1790280514525945 us; error 46 us; skew 500 ppm
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:34.528913 9217 init.cc:377] Logged in from keytab as oryx/127.0.115.129@KRBTEST.COM (short username oryx)
I20260924 20:08:34.529776 9217 webserver.cc:460] Webserver started at http://127.0.115.129:37209/ using document root <none> and password file <none>
I20260924 20:08:34.530364 9217 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:34.530471 9217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:34.534993 9217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:08:34.537348 9251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.538743 9217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:08:34.538884 9217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "9e697b81c7554a7798d0947e8a3b5afd"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "d33d6322a3bed4d88392f82abdb30c84"
server_key_iv: "a0771e77e09bafba5f783a8b28179aa7"
server_key_version: "encryptionkey@0"
I20260924 20:08:34.539479 9217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:34.562099 9217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:34.565550 9217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:34.565766 9217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:34.566402 9217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:34.567276 9217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:34.567356 9217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.567448 9217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:34.567499 9217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.576800 9217 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:40635
I20260924 20:08:34.576818 9364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:40635 every 8 connection(s)
I20260924 20:08:34.577831 9217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:34.580071 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9217
I20260924 20:08:34.580586 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:36939
--local_ip_for_outbound_sockets=127.0.115.130
--tserver_master_addrs=127.0.115.190:38589
--webserver_port=33989
--webserver_interface=127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.591984 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.580198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:40189)
0924 20:08:34.580691 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.580734 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.582127 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.583349 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.583358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.583792 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.584336 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.584351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.585668 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.585673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.586587 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.586597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.587448 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.587452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.587656 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.588323 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.588341 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.590910 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":151,"threads_started":1}
W20260924 20:08:34.591981 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.579986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:40189 (local address 127.0.115.190:38589)
0924 20:08:34.580478 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.580502 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.580865 (+ 363us) server_negotiation.cc:407] Connection header received
0924 20:08:34.582587 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.582598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.582913 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.583265 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.584518 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.585521 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.586725 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.587267 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.587462 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.591142 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:40189: BlockingRecv error: recv got EOF from 127.0.115.129:40189 (error 108)
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":130,"threads_started":1}
W20260924 20:08:34.592728 9365 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:38589 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.115.190:38589: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.598407 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.593220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:52561)
0924 20:08:34.593360 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.593383 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.593453 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.593781 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.593785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.593942 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.594030 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.594035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.594792 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.594797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.595345 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.595353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.595816 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.595819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.595913 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.596453 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.596465 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.598257 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 20:08:34.598495 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.593315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:52561 (local address 127.0.115.190:38589)
0924 20:08:34.593583 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.593587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.593597 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:08:34.593627 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.593629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.593674 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.593764 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.594141 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.594676 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.595463 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.595697 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.595803 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.598379 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:52561: BlockingRecv error: recv got EOF from 127.0.115.129:52561 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.604487 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.599157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:59423)
0924 20:08:34.599329 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.599343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.599411 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.599649 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.599652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.599815 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.599914 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.599922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.600639 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.600644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.601489 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.601500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.601961 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.601965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.602046 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.602489 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.602506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.604363 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 20:08:34.604631 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.599255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:59423 (local address 127.0.115.190:38589)
0924 20:08:34.599367 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.599371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.599383 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:08:34.599495 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.599497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.599534 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.599596 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.600049 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.600530 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.601604 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.601852 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.601927 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.604487 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:59423: BlockingRecv error: recv got EOF from 127.0.115.129:59423 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 20:08:34.604794 9365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:08:34.697474 9369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:34.697741 9369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:34.697785 9369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:34.701743 9369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:34.701804 9369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:34.701861 9369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:34.707188 9369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.130.keytab
--principal=oryx/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:36939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=33989
--tserver_master_addrs=127.0.115.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:34.708245 9369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:34.709278 9369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:34.719202 9375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.719208 9377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:34.719270 9369 server_base.cc:1034] running on GCE node
W20260924 20:08:34.719506 9374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:34.720007 9369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:34.720646 9369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:34.721802 9369 hybrid_clock.cc:648] HybridClock initialized: now 1790280514721756 us; error 59 us; skew 500 ppm
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:34.724748 9369 init.cc:377] Logged in from keytab as oryx/127.0.115.130@KRBTEST.COM (short username oryx)
I20260924 20:08:34.725687 9369 webserver.cc:460] Webserver started at http://127.0.115.130:33989/ using document root <none> and password file <none>
I20260924 20:08:34.726253 9369 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:34.726322 9369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:34.730379 9369 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:08:34.732664 9384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.734047 9369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:34.734192 9369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "db1da0bfc7794144a6308fc5cf4919d4"
format_stamp: "Formatted at 2026-09-24 20:08:33 on dist-test-slave-2pm7"
server_key: "a012a262c0b17b0526df049aa1edc98a"
server_key_iv: "7d837bea864020318367e9055b24a760"
server_key_version: "encryptionkey@0"
I20260924 20:08:34.734750 9369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:34.754642 9369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:34.757798 9369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:34.758008 9369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:34.758626 9369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:34.759481 9369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:34.759550 9369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.759624 9369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:34.759673 9369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.769308 9369 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:36939
I20260924 20:08:34.769330 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:36939 every 8 connection(s)
I20260924 20:08:34.770335 9369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:34.777796 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9369
I20260924 20:08:34.778312 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:34113
--local_ip_for_outbound_sockets=127.0.115.131
--tserver_master_addrs=127.0.115.190:38589
--webserver_port=37903
--webserver_interface=127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:08:34.781942 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.772168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:48987 (local address 127.0.115.190:38589)
0924 20:08:34.772324 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.772330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.773275 (+ 945us) server_negotiation.cc:407] Connection header received
0924 20:08:34.774608 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.774611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.774661 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.774734 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.775901 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.776446 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.777612 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.777823 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.777959 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.781816 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:48987: BlockingRecv error: recv got EOF from 127.0.115.130:48987 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:08:34.782611 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.772645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:48987)
0924 20:08:34.773120 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.773162 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.774404 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.774909 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.774915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.775252 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.775741 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.775754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.776569 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.776571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.777450 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.777460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.778010 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.778015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.778244 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.778927 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.778954 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.781683 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
W20260924 20:08:34.783107 9498 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:38589 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.115.190:38589: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.788676 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.783535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:43661)
0924 20:08:34.783649 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.783665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.783745 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.784021 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.784024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.784178 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.784259 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.784263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.785001 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.785006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.785543 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.785551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.786007 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.786011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.786111 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.786612 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.786623 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.788540 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 20:08:34.788800 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.783604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:43661 (local address 127.0.115.190:38589)
0924 20:08:34.783722 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.783726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.783739 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:34.783847 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.783850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.783893 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.784009 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.784383 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.784891 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.785666 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.785897 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.786005 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.788658 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:43661: BlockingRecv error: recv got EOF from 127.0.115.130:43661 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.794430 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.789374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:44261)
0924 20:08:34.789510 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.789524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.789607 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.789859 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.789862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.790019 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.790100 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.790105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.790852 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.790857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.791382 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.791390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.791832 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.791836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.791914 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.792419 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.792430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.794298 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 20:08:34.794523 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.789524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:44261 (local address 127.0.115.190:38589)
0924 20:08:34.789653 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.789660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.789674 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:34.789716 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.789718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.789750 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.789843 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.790213 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.790741 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.791503 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.791723 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.791830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.794404 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:44261: BlockingRecv error: recv got EOF from 127.0.115.130:44261 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 20:08:34.795106 9498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:08:34.896409 9501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:34.896720 9501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:34.896798 9501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:34.900971 9501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:34.901041 9501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:34.901162 9501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:34.906682 9501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/krb5kdc/oryx_127.0.115.131.keytab
--principal=oryx/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:34113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=37903
--tserver_master_addrs=127.0.115.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:34.907745 9501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:34.908833 9501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:34.918695 9507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.918685 9509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:34.918748 9506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:34.919070 9501 server_base.cc:1034] running on GCE node
I20260924 20:08:34.919446 9501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:34.920225 9501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:34.921412 9501 hybrid_clock.cc:648] HybridClock initialized: now 1790280514921371 us; error 61 us; skew 500 ppm
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:34.924672 9501 init.cc:377] Logged in from keytab as oryx/127.0.115.131@KRBTEST.COM (short username oryx)
I20260924 20:08:34.925562 9501 webserver.cc:460] Webserver started at http://127.0.115.131:37903/ using document root <none> and password file <none>
I20260924 20:08:34.926133 9501 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:34.926209 9501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:34.930127 9501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 20:08:34.932243 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.933499 9501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 20:08:34.933629 9501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "ab5a4205cbd4433d973fe3b11d1b9323"
format_stamp: "Formatted at 2026-09-24 20:08:34 on dist-test-slave-2pm7"
server_key: "1928b6dffca27d1d4939a23873597e58"
server_key_iv: "7b167bfebcf8b3192f70e2a946130dda"
server_key_version: "encryptionkey@0"
I20260924 20:08:34.934187 9501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:34.946409 9501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:34.949662 9501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:34.949874 9501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:34.950515 9501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:34.951436 9501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:34.951493 9501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.951597 9501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:34.951640 9501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:34.961792 9501 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:34113
I20260924 20:08:34.961813 9629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:34113 every 8 connection(s)
I20260924 20:08:34.962930 9501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:34.965488 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9501
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.975623 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.966609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.0.115.190:38589)
0924 20:08:34.966834 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.966838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.966851 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:34.966892 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.966894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.966934 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.967011 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.968189 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.968755 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.969663 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.969907 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.970021 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.971488 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:34.971494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:34.971508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:34.971535 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:34.973454 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:34.973952 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:34.973957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:34.973962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:34.974080 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:34.974369 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:34.974372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:34.974374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:34.974754 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:34.974965 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:34.975246 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 20:08:34.975487 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
W20260924 20:08:34.976370 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.964988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:35899 (local address 127.0.115.190:38589)
0924 20:08:34.965143 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.965148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.966045 (+ 897us) server_negotiation.cc:407] Connection header received
0924 20:08:34.968647 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.968651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.968712 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.968802 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.970089 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.970659 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.972037 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.972379 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.972499 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.976268 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:35899: BlockingRecv error: recv got EOF from 127.0.115.131:35899 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:08:34.977219 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.965301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:35899)
0924 20:08:34.965852 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.965917 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.968383 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.968996 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.969005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.969391 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.969924 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.969938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.970826 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.970830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.971859 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.971873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.972684 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.972695 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.972883 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.973466 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.973486 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.976181 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":115,"threads_started":1}
W20260924 20:08:34.977699 9630 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:38589 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.115.190:38589: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.983042 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.978157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:40081)
0924 20:08:34.978265 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.978278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.978353 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.978583 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.978586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.978760 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.978873 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.978879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.979590 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.979594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.980232 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.980240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.980673 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.980676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.980785 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.981218 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.981228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.982901 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 20:08:34.983119 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.978232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:40081 (local address 127.0.115.190:38589)
0924 20:08:34.978354 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.978359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.978373 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:34.978409 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.978411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.978461 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.978540 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.978987 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.979483 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.980346 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.980564 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.980669 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.983007 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:40081: BlockingRecv error: recv got EOF from 127.0.115.131:40081 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:34 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:34.988461 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:34.983719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:43779)
0924 20:08:34.983880 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:34.983895 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:34.983990 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:34.984216 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:34.984219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:34.984375 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:34.984439 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.984444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.985133 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.985138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.985646 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:34.985654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.986083 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.986087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:34.986164 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.986556 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:34.986566 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:34.988365 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 20:08:34.988628 9367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:34.983784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:43779 (local address 127.0.115.190:38589)
0924 20:08:34.983974 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:34.983978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:34.983991 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:34.984077 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:34.984079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:34.984122 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:34.984214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:34.984547 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.985028 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.985763 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:34.985977 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:34.986082 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:34.988500 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:43779: BlockingRecv error: recv got EOF from 127.0.115.131:43779 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 20:08:34.988792 9630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 20:08:35 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:35 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:35.610716 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:35.605398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:44475)
0924 20:08:35.605685 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:35.605698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:35.605789 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:35.606061 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:35.606063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:35.606243 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:35.606339 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:35.606344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.607093 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.607095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:35.607688 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:35.607697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.608119 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.608121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:35.608195 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:35.608773 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:35.608785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:35.610573 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260924 20:08:35.610882 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:35.605502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:44475 (local address 127.0.115.190:38589)
0924 20:08:35.605747 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:35.605751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:35.605764 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:35.605896 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:35.605899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:35.605943 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:35.606017 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:35.606433 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.606965 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.607790 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.608027 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.608103 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:35.610747 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:44475: BlockingRecv error: recv got EOF from 127.0.115.129:44475 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
Sep 24 20:08:35 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:35 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:35.801133 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:35.795728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:43949)
0924 20:08:35.796025 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:35.796039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:35.796132 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:35.796389 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:35.796391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:35.796562 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:35.796656 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:35.796661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.797516 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.797519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:35.798093 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:35.798101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.798542 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.798544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:35.798607 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:35.799150 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:35.799160 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:35.801011 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
W20260924 20:08:35.801354 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:35.795840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:43949 (local address 127.0.115.190:38589)
0924 20:08:35.796054 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:35.796058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:35.796095 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:08:35.796226 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:35.796228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:35.796277 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:35.796356 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:35.796834 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.797418 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.798206 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.798440 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.798523 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:35.801204 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:43949: BlockingRecv error: recv got EOF from 127.0.115.130:43949 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 20:08:35 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:35 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:35.995397 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:35.989612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:34055)
0924 20:08:35.989924 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:35.989943 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:35.990048 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:35.990353 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:35.990356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:35.990546 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:35.990678 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:35.990685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.991421 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.991424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:35.992066 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:35.992076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.992499 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.992502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:35.992567 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:35.993180 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:35.993191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:35.995271 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":114,"threads_started":1}
W20260924 20:08:35.995533 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:35.989773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:34055 (local address 127.0.115.190:38589)
0924 20:08:35.989929 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:35.989933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:35.990018 (+ 85us) server_negotiation.cc:407] Connection header received
0924 20:08:35.990134 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:35.990137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:35.990184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:35.990275 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:35.990796 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.991322 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.992172 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:35.992396 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:35.992474 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:35.995410 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:34055: BlockingRecv error: recv got EOF from 127.0.115.131:34055 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 20:08:36 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:36 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:36.617242 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:36.611680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:40429)
0924 20:08:36.612077 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:36.612091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:36.612188 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:36.612445 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:36.612447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:36.612637 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:36.612736 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:36.612741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.613454 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.613457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:36.614097 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:36.614106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.614556 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.614559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:36.614671 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:36.615166 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:36.615178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:36.617043 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
W20260924 20:08:36.617307 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:36.611851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:40429 (local address 127.0.115.190:38589)
0924 20:08:36.612149 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:36.612153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:36.612166 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:36.612285 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:36.612287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:36.612330 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:36.612404 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:36.612837 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.613363 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.614214 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.614452 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.614532 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:36.617167 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:40429: BlockingRecv error: recv got EOF from 127.0.115.129:40429 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":149,"threads_started":1}
Sep 24 20:08:36 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:36 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:36.807858 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:36.801956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:43533)
0924 20:08:36.802266 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:36.802281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:36.802387 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:36.802790 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:36.802793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:36.802977 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:36.803104 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:36.803109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.804066 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.804069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:36.804645 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:36.804654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.805137 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.805140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:36.805202 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:36.805837 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:36.805850 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:36.807711 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
W20260924 20:08:36.808072 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:36.802137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:43533 (local address 127.0.115.190:38589)
0924 20:08:36.802325 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:36.802329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:36.802344 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:36.802528 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:36.802531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:36.802649 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:36.802728 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:36.803250 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.803865 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.804797 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.805025 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.805116 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:36.807876 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:43533: BlockingRecv error: recv got EOF from 127.0.115.130:43533 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 20:08:36 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:36 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:37.002152 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:36.996266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:58341)
0924 20:08:36.996588 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:36.996602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:36.996699 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:36.996948 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:36.996950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:36.997133 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:36.997241 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:36.997247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.998149 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.998152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:36.998713 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:36.998722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.999173 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.999176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:36.999241 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:36.999837 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:36.999849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:37.001994 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":134,"threads_started":1}
W20260924 20:08:37.002277 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:36.996433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:58341 (local address 127.0.115.190:38589)
0924 20:08:36.996596 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:36.996600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:36.996674 (+ 74us) server_negotiation.cc:407] Connection header received
0924 20:08:36.996783 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:36.996785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:36.996832 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:36.996912 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:36.997345 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.998002 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.998842 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:36.999058 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:36.999137 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:37.002129 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:58341: BlockingRecv error: recv got EOF from 127.0.115.131:58341 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 20:08:37 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:37 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:37.624118 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:37.618211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:33303)
0924 20:08:37.618532 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:37.618549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:37.618672 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:37.618965 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:37.618968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:37.619149 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:37.619258 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:37.619263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.620098 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.620101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:37.620677 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:37.620686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.621249 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.621252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:37.621318 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:37.621929 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:37.621941 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:37.623913 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
W20260924 20:08:37.624269 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:37.618294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:33303 (local address 127.0.115.190:38589)
0924 20:08:37.618566 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:37.618571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:37.618609 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:08:37.618787 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:37.618790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:37.618854 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:37.618938 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:37.619380 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.619963 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.620804 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.621116 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.621231 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:37.624100 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:33303: BlockingRecv error: recv got EOF from 127.0.115.129:33303 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
Sep 24 20:08:37 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:37 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:37.814927 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:37.808988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:32825)
0924 20:08:37.809331 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:37.809351 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:37.809492 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:37.809795 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:37.809797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:37.810061 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:37.810165 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:37.810171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.811053 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.811058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:37.811624 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:37.811632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.812193 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.812197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:37.812280 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:37.812844 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:37.812856 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:37.814780 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":131,"threads_started":1}
W20260924 20:08:37.815105 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:37.809074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:32825 (local address 127.0.115.190:38589)
0924 20:08:37.809251 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:37.809256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:37.809456 (+ 200us) server_negotiation.cc:407] Connection header received
0924 20:08:37.809595 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:37.809598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:37.809657 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:37.809756 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:37.810289 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.810930 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.811765 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:37.812065 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:37.812180 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:37.814937 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:32825: BlockingRecv error: recv got EOF from 127.0.115.130:32825 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 20:08:38 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:38 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:38.008562 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:38.003010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:40487)
0924 20:08:38.003310 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:38.003324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:38.003423 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:38.003629 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:38.003630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:38.003823 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:38.003920 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:38.003925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.004809 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.004812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:38.005397 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:38.005408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.005895 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.005899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:38.005984 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:38.006504 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:38.006517 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:38.008422 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
W20260924 20:08:38.008749 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:38.003146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:40487 (local address 127.0.115.190:38589)
0924 20:08:38.003309 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:38.003313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:38.003433 (+ 120us) server_negotiation.cc:407] Connection header received
0924 20:08:38.003487 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:38.003490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:38.003539 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:38.003614 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:38.004094 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.004633 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.005512 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.005761 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.005841 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:38.008573 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:40487: BlockingRecv error: recv got EOF from 127.0.115.131:40487 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 20:08:38 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:38 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:38.634716 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:38.624979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:42457)
0924 20:08:38.625317 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:38.625331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:38.625430 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:38.625771 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:38.625776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:38.626010 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:38.626114 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:38.626121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.627157 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.627161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:38.627727 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:38.627735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.628204 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.628207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:38.628289 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:38.632466 (+ 4177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:38.632479 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:38.634513 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
W20260924 20:08:38.634851 9648 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:38.625102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:42457 (local address 127.0.115.190:38589)
0924 20:08:38.625379 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:38.625382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:38.625397 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:38.625542 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:38.625545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:38.625603 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:38.625706 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:38.626324 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.627032 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.627830 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.628069 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.628162 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:38.634687 (+ 6525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:42457: BlockingRecv error: recv got EOF from 127.0.115.129:42457 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
Sep 24 20:08:38 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:38 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:38.821148 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:38.815698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:46003)
0924 20:08:38.816048 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:38.816061 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:38.816174 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:38.816403 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:38.816405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:38.816578 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:38.816687 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:38.816692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.817432 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.817435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:38.818001 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:38.818010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.818429 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.818431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:38.818498 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:38.818997 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:38.819010 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:38.821017 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":96,"threads_started":1}
W20260924 20:08:38.821324 9648 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:38.816027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:46003 (local address 127.0.115.190:38589)
0924 20:08:38.816168 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:38.816172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:38.816186 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:38.816259 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:38.816261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:38.816303 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:38.816361 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:38.816794 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.817312 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.818112 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:38.818332 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:38.818406 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:38.821163 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:46003: BlockingRecv error: recv got EOF from 127.0.115.130:46003 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 20:08:39 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:39 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:39.014850 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:39.009441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:58285)
0924 20:08:39.009722 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:39.009736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:39.009838 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:39.010080 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:39.010082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:39.010274 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:39.010367 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:39.010373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.011167 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.011170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:39.011801 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:39.011811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.012294 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.012297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:39.012368 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:39.012843 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:39.012855 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:39.014732 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
W20260924 20:08:39.014992 9648 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:39.009501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:58285 (local address 127.0.115.190:38589)
0924 20:08:39.009656 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:39.009660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:39.009809 (+ 149us) server_negotiation.cc:407] Connection header received
0924 20:08:39.009921 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:39.009924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:39.009970 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:39.010037 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:39.010471 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.011035 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.011930 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.012181 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.012262 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:39.014871 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:58285: BlockingRecv error: recv got EOF from 127.0.115.131:58285 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 20:08:39 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:39 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:39.641420 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:39.635621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:48777)
0924 20:08:39.635936 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:39.635967 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:39.636051 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:39.636308 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:39.636310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:39.636488 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:39.636581 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:39.636586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.637416 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.637419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:39.638043 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:39.638052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.638574 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.638577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:39.638662 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:39.639183 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:39.639198 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:39.641303 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
W20260924 20:08:39.641645 9652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:39.635770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:48777 (local address 127.0.115.190:38589)
0924 20:08:39.636040 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:39.636044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:39.636058 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:39.636137 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:39.636139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:39.636197 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:39.636269 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:39.636690 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.637262 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.638144 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.638454 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.638552 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:39.641494 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:48777: BlockingRecv error: recv got EOF from 127.0.115.129:48777 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
Sep 24 20:08:39 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:39 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:39.827910 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:39.822189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:39087)
0924 20:08:39.822469 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:39.822483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:39.822568 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:39.822902 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:39.822904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:39.823098 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:39.823204 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:39.823210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.824115 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.824118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:39.824695 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:39.824703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.825126 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.825129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:39.825199 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:39.825664 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:39.825678 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:39.827730 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
W20260924 20:08:39.827996 9652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:39.822353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:39087 (local address 127.0.115.190:38589)
0924 20:08:39.822505 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:39.822514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:39.822529 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:39.822713 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:39.822718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:39.822780 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:39.822856 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:39.823307 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.824010 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.824808 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:39.825026 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:39.825105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:39.827858 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:39087: BlockingRecv error: recv got EOF from 127.0.115.130:39087 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 20:08:40 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:40 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:40.021956 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:40.016002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:39375)
0924 20:08:40.016382 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:40.016395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:40.016529 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:40.016827 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:40.016829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:40.017035 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:40.017124 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:40.017130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.017910 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.017915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:40.018490 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:40.018498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.018973 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.018977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:40.019060 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:40.019539 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:40.019552 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:40.021744 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
W20260924 20:08:40.022024 9652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:40.016059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:39375 (local address 127.0.115.190:38589)
0924 20:08:40.016226 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:40.016232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:40.016475 (+ 243us) server_negotiation.cc:407] Connection header received
0924 20:08:40.016631 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:40.016635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:40.016709 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:40.016799 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:40.017244 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.017797 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.018612 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.018859 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.018969 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:40.021900 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:39375: BlockingRecv error: recv got EOF from 127.0.115.131:39375 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 20:08:40 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:40 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:40.648571 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:40.642272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:55085)
0924 20:08:40.642557 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:40.642571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:40.642676 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:40.643023 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:40.643026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:40.643317 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:40.643429 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:40.643437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.644520 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.644526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:40.645204 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:40.645216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.645711 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.645714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:40.645782 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:40.646371 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:40.646389 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:40.648435 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
W20260924 20:08:40.648792 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:40.642446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:55085 (local address 127.0.115.190:38589)
0924 20:08:40.642740 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:40.642743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:40.642758 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:08:40.642803 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:40.642805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:40.642875 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:40.642985 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:40.643544 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.644396 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.645338 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.645575 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.645661 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:40.648604 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:55085: BlockingRecv error: recv got EOF from 127.0.115.129:55085 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
Sep 24 20:08:40 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:40 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:40.835474 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:40.829012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:57037)
0924 20:08:40.829367 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:40.829382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:40.829535 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:40.829820 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:40.829823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:40.830037 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:40.830129 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:40.830134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.831212 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.831216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:40.832133 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:40.832143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.832783 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.832787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:40.832885 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:40.833389 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:40.833405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:40.835358 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":91,"threads_started":1}
W20260924 20:08:40.835665 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:40.829168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:57037 (local address 127.0.115.190:38589)
0924 20:08:40.829342 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:40.829347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:40.829468 (+ 121us) server_negotiation.cc:407] Connection header received
0924 20:08:40.829648 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:40.829656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:40.829703 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:40.829790 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:40.830288 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.831063 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.832328 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:40.832665 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:40.832773 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:40.835503 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:57037: BlockingRecv error: recv got EOF from 127.0.115.130:57037 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 20:08:41 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:41 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:41.028285 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:41.022926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:33767)
0924 20:08:41.023246 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:41.023261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:41.023364 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:41.023670 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:41.023674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:41.023858 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:41.023983 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:41.023989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.024720 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.024722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:41.025273 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:41.025281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.025721 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.025724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:41.025786 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:41.026272 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:41.026283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:41.028170 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
W20260924 20:08:41.028483 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:41.022968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:33767 (local address 127.0.115.190:38589)
0924 20:08:41.023116 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:41.023121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:41.023339 (+ 218us) server_negotiation.cc:407] Connection header received
0924 20:08:41.023493 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:41.023498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:41.023554 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:41.023620 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:41.024084 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.024621 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.025384 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.025617 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.025698 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:41.028318 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:33767: BlockingRecv error: recv got EOF from 127.0.115.131:33767 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 20:08:41 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:41 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:41.654922 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:41.649468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:39225)
0924 20:08:41.649895 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:41.649918 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:41.650026 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:41.650300 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:41.650303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:41.650486 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:41.650580 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:41.650585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.651352 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.651354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:41.652025 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:41.652043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.652450 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.652452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:41.652516 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:41.653057 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:41.653068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:41.654805 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":88,"threads_started":1}
W20260924 20:08:41.655086 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:41.649545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:39225 (local address 127.0.115.190:38589)
0924 20:08:41.649924 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:41.649927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:41.649991 (+ 64us) server_negotiation.cc:407] Connection header received
0924 20:08:41.650106 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:41.650109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:41.650153 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:41.650230 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:41.650729 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.651256 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.652135 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.652358 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.652433 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:41.654940 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:39225: BlockingRecv error: recv got EOF from 127.0.115.129:39225 (error 108)
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":67,"threads_started":1}
Sep 24 20:08:41 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:41 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:41.842360 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:41.836272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:48863)
0924 20:08:41.836622 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:41.836639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:41.836769 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:41.837095 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:41.837098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:41.837315 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:41.837443 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:41.837451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.838415 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.838418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:41.839159 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:41.839169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.839607 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.839609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:41.839668 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:41.840281 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:41.840293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:41.842233 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":86,"threads_started":1}
W20260924 20:08:41.842480 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:41.836680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:48863 (local address 127.0.115.190:38589)
0924 20:08:41.836840 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:41.836845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:41.836859 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:41.836901 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:41.836904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:41.836953 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:41.837038 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:41.837579 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.838317 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.839279 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:41.839500 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:41.839592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:41.842365 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:48863: BlockingRecv error: recv got EOF from 127.0.115.130:48863 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 20:08:42 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:42 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:42.034811 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:42.029161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:49797)
0924 20:08:42.029483 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:42.029497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:42.029600 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:42.029837 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:42.029839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:42.030017 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:42.030109 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:42.030115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.030831 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.030833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:42.031381 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:42.031388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.031886 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.031890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:42.032011 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:42.032675 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:42.032690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:42.034693 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
W20260924 20:08:42.034945 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:42.029282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:49797 (local address 127.0.115.190:38589)
0924 20:08:42.029423 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:42.029427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:42.029567 (+ 140us) server_negotiation.cc:407] Connection header received
0924 20:08:42.029668 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:42.029671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:42.029718 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:42.029802 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:42.030212 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.030727 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.031512 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.031749 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.031836 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:42.034824 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:49797: BlockingRecv error: recv got EOF from 127.0.115.131:49797 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 20:08:42 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:42 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:42.661648 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:42.655900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:53269)
0924 20:08:42.656221 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:42.656235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:42.656327 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:42.656618 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:42.656621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:42.656808 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:42.656913 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:42.656919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.657834 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.657838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:42.658415 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:42.658448 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.658957 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.658960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:42.659025 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:42.659579 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:42.659591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:42.661465 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":134,"threads_started":1}
W20260924 20:08:42.661754 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:42.656063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:53269 (local address 127.0.115.190:38589)
0924 20:08:42.656302 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:42.656305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:42.656319 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:42.656407 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:42.656410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:42.656453 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:42.656536 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:42.657025 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.657651 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.658553 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.658868 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.658948 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:42.661592 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:53269: BlockingRecv error: recv got EOF from 127.0.115.129:53269 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
Sep 24 20:08:42 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:42 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:42.849170 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:42.843319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:42787)
0924 20:08:42.843618 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:42.843640 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:42.843756 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:42.844063 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:42.844066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:42.844279 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:42.844371 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:42.844376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.845327 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.845332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:42.845910 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:42.845918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.846399 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.846403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:42.846486 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:42.847031 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:42.847052 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:42.848981 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":80,"threads_started":1}
W20260924 20:08:42.849419 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:42.843443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:42787 (local address 127.0.115.190:38589)
0924 20:08:42.843610 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:42.843615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:42.843698 (+ 83us) server_negotiation.cc:407] Connection header received
0924 20:08:42.843879 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:42.843882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:42.843956 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:42.844044 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:42.844488 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.845206 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.846031 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:42.846282 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:42.846394 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:42.849154 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:42787: BlockingRecv error: recv got EOF from 127.0.115.130:42787 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 20:08:43 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:43 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:43.042562 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:43.035743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:41977)
0924 20:08:43.036115 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:43.036130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:43.036296 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:43.036604 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:43.036608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:43.036903 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:43.037048 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:43.037056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.038137 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.038141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:43.038772 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:43.038782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.039274 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.039278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:43.039364 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:43.040009 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:43.040031 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:43.042343 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":89,"threads_started":1}
W20260924 20:08:43.042651 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:43.035913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:41977 (local address 127.0.115.190:38589)
0924 20:08:43.036080 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:43.036085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:43.036234 (+ 149us) server_negotiation.cc:407] Connection header received
0924 20:08:43.036419 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:43.036422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:43.036472 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:43.036544 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:43.037201 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.038024 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.038899 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.039155 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.039266 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:43.042527 (+ 3261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:41977: BlockingRecv error: recv got EOF from 127.0.115.131:41977 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 20:08:43 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:43 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:43.676586 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:43.662401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:48285)
0924 20:08:43.662697 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:43.662714 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:43.662807 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:43.663052 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:43.663054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:43.663231 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:43.663326 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:43.663331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.664174 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.664176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:43.664758 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:43.664766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.665178 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.665180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:43.665255 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:43.674495 (+ 9240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:43.674509 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:43.676464 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
W20260924 20:08:43.676810 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:43.662492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:48285 (local address 127.0.115.190:38589)
0924 20:08:43.662772 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:43.662776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:43.662789 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:43.662894 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:43.662896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:43.662939 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:43.663012 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:43.663424 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.664065 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.664862 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.665083 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.665163 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:43.676609 (+ 11446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:48285: BlockingRecv error: recv got EOF from 127.0.115.129:48285 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
Sep 24 20:08:43 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:43 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:43.855716 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:43.850017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:58017)
0924 20:08:43.850335 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:43.850350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:43.850491 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:43.850768 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:43.850771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:43.850978 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:43.851069 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:43.851075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.851885 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.851889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:43.852502 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:43.852511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.852988 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.852992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:43.853076 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:43.853578 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:43.853590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:43.855529 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
W20260924 20:08:43.855805 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:43.850085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:58017 (local address 127.0.115.190:38589)
0924 20:08:43.850284 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:43.850290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:43.850432 (+ 142us) server_negotiation.cc:407] Connection header received
0924 20:08:43.850600 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:43.850603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:43.850659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:43.850746 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:43.851197 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.851765 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.852627 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:43.852872 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:43.852984 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:43.855674 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:58017: BlockingRecv error: recv got EOF from 127.0.115.130:58017 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 20:08:44 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:44 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:44.049296 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:44.043410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:33045)
0924 20:08:44.043733 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:44.043747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:44.043903 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:44.044231 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:44.044234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:44.044449 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:44.044542 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:44.044547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.045353 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.045358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:44.045932 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:44.045939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.046424 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.046428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:44.046517 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:44.047035 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:44.047047 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:44.049175 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
W20260924 20:08:44.049500 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:44.043502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:33045 (local address 127.0.115.190:38589)
0924 20:08:44.043679 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:44.043685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:44.043820 (+ 135us) server_negotiation.cc:407] Connection header received
0924 20:08:44.044039 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:44.044042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:44.044101 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:44.044210 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:44.044665 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.045228 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.046055 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.046294 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.046419 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:44.049325 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:33045: BlockingRecv error: recv got EOF from 127.0.115.131:33045 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 20:08:44 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:44 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:44.682783 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:44.677477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:48865)
0924 20:08:44.677757 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:44.677771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:44.677854 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:44.678110 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:44.678112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:44.678311 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:44.678405 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:44.678411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.679267 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.679271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:44.679836 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:44.679845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.680319 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.680322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:44.680385 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:44.680871 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:44.680883 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:44.682631 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
W20260924 20:08:44.682940 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:44.677527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:48865 (local address 127.0.115.190:38589)
0924 20:08:44.677798 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:44.677802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:44.677816 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:44.677951 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:44.677954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:44.677998 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:44.678077 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:44.678510 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.679151 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.679984 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.680219 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.680301 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:44.682801 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:48865: BlockingRecv error: recv got EOF from 127.0.115.129:48865 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":69,"threads_started":1}
Sep 24 20:08:44 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:44 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:44.862104 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:44.856572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:59617)
0924 20:08:44.856871 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:44.856884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:44.856979 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:44.857212 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:44.857214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:44.857387 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:44.857482 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:44.857487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.858278 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.858281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:44.858856 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:44.858865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.859329 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.859332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:44.859400 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:44.859895 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:44.859906 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:44.861925 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
W20260924 20:08:44.862293 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:44.856702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:59617 (local address 127.0.115.190:38589)
0924 20:08:44.856851 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:44.856855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:44.856936 (+ 81us) server_negotiation.cc:407] Connection header received
0924 20:08:44.857058 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:44.857060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:44.857101 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:44.857169 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:44.857620 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.858169 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.858966 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:44.859201 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:44.859281 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:44.862131 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:59617: BlockingRecv error: recv got EOF from 127.0.115.130:59617 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 20:08:45 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:45 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:45.055836 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:45.050140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:42611)
0924 20:08:45.050428 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:45.050443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:45.050544 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:45.050851 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:45.050854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:45.051039 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:45.051137 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:45.051142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.052046 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.052050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:45.052612 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:45.052621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.053059 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.053062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:45.053127 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:45.053701 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:45.053713 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:45.055673 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
W20260924 20:08:45.056205 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:45.050282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:42611 (local address 127.0.115.190:38589)
0924 20:08:45.050419 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:45.050422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:45.050539 (+ 117us) server_negotiation.cc:407] Connection header received
0924 20:08:45.050652 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:45.050654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:45.050705 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:45.050782 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:45.051261 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.051859 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.052725 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.052950 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.053032 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:45.055859 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:42611: BlockingRecv error: recv got EOF from 127.0.115.131:42611 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 20:08:45 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:45 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:45.689042 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:45.683711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:45129)
0924 20:08:45.684016 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:45.684037 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:45.684151 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:45.684375 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:45.684377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:45.684549 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:45.684656 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:45.684663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.685486 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.685489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:45.686074 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:45.686083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.686536 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.686539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:45.686602 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:45.687098 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:45.687110 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:45.688925 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260924 20:08:45.689208 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:45.683742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:45129 (local address 127.0.115.190:38589)
0924 20:08:45.684022 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:45.684027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:45.684095 (+ 68us) server_negotiation.cc:407] Connection header received
0924 20:08:45.684217 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:45.684219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:45.684261 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:45.684336 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:45.684809 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.685366 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.686202 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.686432 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.686513 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:45.689058 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:45129: BlockingRecv error: recv got EOF from 127.0.115.129:45129 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
Sep 24 20:08:45 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:45 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:45.869057 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:45.863010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:56613)
0924 20:08:45.863350 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:45.863364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:45.863512 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:45.863809 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:45.863812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:45.864081 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:45.864184 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:45.864190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.864954 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.864958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:45.865520 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:45.865528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.866007 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.866011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:45.866098 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:45.866685 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:45.866698 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:45.868848 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
W20260924 20:08:45.869130 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:45.863135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:56613 (local address 127.0.115.190:38589)
0924 20:08:45.863311 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:45.863316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:45.863445 (+ 129us) server_negotiation.cc:407] Connection header received
0924 20:08:45.863622 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:45.863626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:45.863679 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:45.863778 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:45.864300 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.864843 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.865641 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:45.865893 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:45.866003 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:45.868994 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:56613: BlockingRecv error: recv got EOF from 127.0.115.130:56613 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 20:08:46 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:46 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:46.062983 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:46.057014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:59075)
0924 20:08:46.057336 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:46.057351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:46.057466 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:46.057727 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:46.057730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:46.057921 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:46.058022 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:46.058028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.058866 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.058869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:46.059506 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:46.059516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.060012 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.060015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:46.060104 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:46.060659 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:46.060680 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:46.062838 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
W20260924 20:08:46.063064 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:46.057103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:59075 (local address 127.0.115.190:38589)
0924 20:08:46.057311 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:46.057315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:46.057430 (+ 115us) server_negotiation.cc:407] Connection header received
0924 20:08:46.057557 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:46.057559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:46.057606 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:46.057676 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:46.058140 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.058706 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.059628 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.059856 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.059935 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:46.062969 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:59075: BlockingRecv error: recv got EOF from 127.0.115.131:59075 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 20:08:46 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:46 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:46.695683 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:46.689916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:60429)
0924 20:08:46.690227 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:46.690240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:46.690335 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:46.690563 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:46.690565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:46.690777 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:46.690876 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:46.690882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.691647 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.691651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:46.692306 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:46.692317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.692775 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.692778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:46.692844 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:46.693423 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:46.693435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:46.695500 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":128,"threads_started":1}
W20260924 20:08:46.695875 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:46.689976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:60429 (local address 127.0.115.190:38589)
0924 20:08:46.690233 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:46.690237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:46.690286 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:08:46.690417 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:46.690419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:46.690463 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:46.690541 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:46.690986 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.691537 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.692416 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.692663 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.692746 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:46.695722 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:60429: BlockingRecv error: recv got EOF from 127.0.115.129:60429 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
Sep 24 20:08:46 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:46 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:08:46.876642 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:46.870277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:57387 (local address 127.0.115.190:38589)
0924 20:08:46.870431 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:46.870436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:46.870502 (+ 66us) server_negotiation.cc:407] Connection header received
0924 20:08:46.870631 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:46.870634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:46.870683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:46.870767 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:46.871279 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.871889 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.872856 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.873249 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.873385 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:46.876537 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:57387: BlockingRecv error: recv got EOF from 127.0.115.130:57387 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:08:46.876699 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:46.870108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:57387)
0924 20:08:46.870397 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:46.870415 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:46.870541 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:46.870839 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:46.870842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:46.871026 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:46.871131 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:46.871137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.872132 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.872136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:46.872724 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:46.872733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:46.873396 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:46.873401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:46.873499 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:46.874270 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:46.874284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:46.876412 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
Sep 24 20:08:47 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:47 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:47.069613 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:47.063894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:46013)
0924 20:08:47.064202 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:47.064223 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:47.064358 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:47.064635 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:47.064638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:47.064853 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:47.064948 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:47.064954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.065758 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.065763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:47.066334 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:47.066342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.066835 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.066839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:47.066933 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:47.067433 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:47.067445 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:47.069439 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
W20260924 20:08:47.069705 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:47.064013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:46013 (local address 127.0.115.190:38589)
0924 20:08:47.064172 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:47.064177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:47.064294 (+ 117us) server_negotiation.cc:407] Connection header received
0924 20:08:47.064459 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:47.064464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:47.064522 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:47.064608 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:47.065068 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.065641 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.066458 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.066708 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.066818 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:47.069572 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:46013: BlockingRecv error: recv got EOF from 127.0.115.131:46013 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 20:08:47 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:47 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:47.702307 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:47.696572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:56665)
0924 20:08:47.696915 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:47.696930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:47.697031 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:47.697305 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:47.697308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:47.697513 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:47.697679 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:47.697685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.698457 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.698460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:47.699159 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:47.699170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.699672 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.699675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:47.699746 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:47.700256 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:47.700269 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:47.702180 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
W20260924 20:08:47.702560 9684 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:47.696651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:56665 (local address 127.0.115.190:38589)
0924 20:08:47.696957 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:47.696961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:47.696978 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:08:47.697124 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:47.697127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:47.697185 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:47.697259 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:47.697791 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.698330 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.699275 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.699496 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.699571 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:47.702414 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:56665: BlockingRecv error: recv got EOF from 127.0.115.129:56665 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
Sep 24 20:08:47 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:47 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:47.883348 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:47.877547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:52931)
0924 20:08:47.877947 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:47.877960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:47.878064 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:47.878350 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:47.878353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:47.878535 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:47.878636 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:47.878642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.879361 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.879364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:47.879916 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:47.879923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.880545 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.880548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:47.880635 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:47.881244 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:47.881262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:47.883164 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":108,"threads_started":1}
W20260924 20:08:47.883400 9684 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:47.877724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:52931 (local address 127.0.115.190:38589)
0924 20:08:47.877858 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:47.877862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:47.878024 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:08:47.878161 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:47.878164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:47.878207 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:47.878278 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:47.878744 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.879264 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.880076 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:47.880403 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:47.880537 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:47.883288 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:52931: BlockingRecv error: recv got EOF from 127.0.115.130:52931 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 20:08:48 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:48 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:48.076074 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:48.070441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:49569)
0924 20:08:48.070780 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:48.070792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:48.070887 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:48.071117 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:48.071119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:48.071303 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:48.071395 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:48.071400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.072249 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.072252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:48.072843 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:48.072852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.073306 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.073309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:48.073374 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:48.073914 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:48.073926 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:48.075852 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":79,"threads_started":1}
W20260924 20:08:48.076119 9684 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:48.070582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:49569 (local address 127.0.115.190:38589)
0924 20:08:48.070743 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:48.070748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:48.070845 (+ 97us) server_negotiation.cc:407] Connection header received
0924 20:08:48.070969 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:48.070971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:48.071017 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:48.071078 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:48.071507 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.072144 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.072963 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.073198 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.073280 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:48.076008 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:49569: BlockingRecv error: recv got EOF from 127.0.115.131:49569 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 20:08:48 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:48 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:48.714188 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:48.703224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:34439)
0924 20:08:48.703505 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:48.703521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:48.703645 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:48.704053 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:48.704055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:48.704262 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:48.704360 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:48.704366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.705434 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.705438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:48.706333 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:48.706343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.706844 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.706847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:48.706911 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:48.711663 (+ 4752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:48.711678 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:48.713972 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
W20260924 20:08:48.714270 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:48.703389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:34439 (local address 127.0.115.190:38589)
0924 20:08:48.703777 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:48.703781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:48.703794 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:08:48.703834 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:48.703836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:48.703878 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:48.704027 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:48.704482 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.705299 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.706466 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.706742 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.706832 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:48.714123 (+ 7291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:34439: BlockingRecv error: recv got EOF from 127.0.115.129:34439 (error 108)
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
Sep 24 20:08:48 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:48 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:48.889896 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:48.884235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:57599)
0924 20:08:48.884530 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:48.884542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:48.884679 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:48.884950 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:48.884953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:48.885170 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:48.885269 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:48.885275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.886123 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.886126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:48.886713 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:48.886722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.887156 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.887160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:48.887227 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:48.887781 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:48.887794 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:48.889771 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":100,"threads_started":1}
W20260924 20:08:48.890072 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:48.884282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:57599 (local address 127.0.115.190:38589)
0924 20:08:48.884425 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:48.884430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:48.884616 (+ 186us) server_negotiation.cc:407] Connection header received
0924 20:08:48.884787 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:48.884791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:48.884844 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:48.884927 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:48.885422 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.885991 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.886821 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:48.887039 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:48.887115 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:48.889932 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:57599: BlockingRecv error: recv got EOF from 127.0.115.130:57599 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 20:08:49 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:49 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.131@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:49.082922 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:49.077087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.131:41737)
0924 20:08:49.077529 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:49.077542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:49.077713 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:49.078000 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:49.078003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:49.078214 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:49.078313 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:49.078319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.079077 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.079082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:49.079674 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:49.079682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.080210 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.080213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:49.080301 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:49.080831 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:49.080843 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:49.082732 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":115,"threads_started":1}
W20260924 20:08:49.082983 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:49.077181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:41737 (local address 127.0.115.190:38589)
0924 20:08:49.077461 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:49.077466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:49.077618 (+ 152us) server_negotiation.cc:407] Connection header received
0924 20:08:49.077843 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:49.077845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:49.077890 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:49.077974 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:49.078437 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.078965 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.079798 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.080094 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.080205 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:49.082864 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.131:41737: BlockingRecv error: recv got EOF from 127.0.115.131:41737 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 20:08:49 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:49 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.129@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:49.721484 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:49.715225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.129:58951)
0924 20:08:49.715545 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:49.715560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:49.715670 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:49.715966 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:49.715969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:49.716172 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:49.716286 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:49.716292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.717096 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.717099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:49.718027 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:49.718039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.718549 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.718552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:49.718636 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:49.719135 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:49.719165 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:49.721292 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":84,"threads_started":1}
W20260924 20:08:49.721673 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:49.715370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:58951 (local address 127.0.115.190:38589)
0924 20:08:49.715612 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:49.715616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:49.715630 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:08:49.715773 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:49.715776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:49.715822 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:49.715899 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:49.716400 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.716968 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.718175 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.718424 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.718541 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:49.721448 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.129:58951: BlockingRecv error: recv got EOF from 127.0.115.129:58951 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":60,"threads_started":1}
Sep 24 20:08:49 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:08:49 dist-test-slave-2pm7 krb5kdc[8634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.115.130@KRBTEST.COM for oryx/127.0.115.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:08:49.896806 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:49.890930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:38589 (local address 127.0.115.130:51133)
0924 20:08:49.891208 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:49.891222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:49.891322 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:49.891563 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:49.891565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:49.891754 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:49.891847 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:49.891853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.892719 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.892722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:49.893316 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:49.893325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.893846 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.893849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:49.893916 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:49.894444 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:49.894455 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:49.896621 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.115.190:38589: Server oryx/127.0.115.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
W20260924 20:08:49.896893 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:49.891006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:51133 (local address 127.0.115.190:38589)
0924 20:08:49.891169 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:49.891176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:49.891296 (+ 120us) server_negotiation.cc:407] Connection header received
0924 20:08:49.891406 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:49.891409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:49.891454 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:49.891520 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:49.892043 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.892576 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.893471 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:49.893723 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:49.893819 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:49.896770 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.115.130:51133: BlockingRecv error: recv got EOF from 127.0.115.130:51133 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:08:49.967528 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9217
I20260924 20:08:49.973613 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9369
I20260924 20:08:49.979319 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9501
I20260924 20:08:49.984447 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9147
2026-09-24T20:08:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18045 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T20:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:50Z Disabled control of system clock
I20260924 20:08:50.015115 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:41259
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 20:08:50.150333 9697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:50.150589 9697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:50.150624 9697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:50.154788 9697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:50.154844 9697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:50.154860 9697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:50.154876 9697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:50.154891 9697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:50.159762 9697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:41259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:50.161018 9697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:50.162518 9697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:50.171769 9705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:50.171717 9702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:50.171720 9703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:50.171762 9697 server_base.cc:1034] running on GCE node
I20260924 20:08:50.172623 9697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:50.173771 9697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:50.174952 9697 hybrid_clock.cc:648] HybridClock initialized: now 1790280530174925 us; error 38 us; skew 500 ppm
I20260924 20:08:50.177026 9697 webserver.cc:460] Webserver started at http://127.0.115.190:38671/ using document root <none> and password file <none>
I20260924 20:08:50.177600 9697 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:50.177662 9697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:50.177830 9697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:50.180496 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "2c422607d0494c3aaf8c99dea7cdccfe"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "ea9bb597d5d80c9c3d1aa849f2040863"
server_key_iv: "e47f565776c86388744137aab8c555e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.181334 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "2c422607d0494c3aaf8c99dea7cdccfe"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "ea9bb597d5d80c9c3d1aa849f2040863"
server_key_iv: "e47f565776c86388744137aab8c555e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.186336 9697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 20:08:50.190305 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.191843 9697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 20:08:50.192035 9697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "2c422607d0494c3aaf8c99dea7cdccfe"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "ea9bb597d5d80c9c3d1aa849f2040863"
server_key_iv: "e47f565776c86388744137aab8c555e4"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.192179 9697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:50.220667 9697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:50.221661 9697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:50.221884 9697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:50.230826 9697 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:41259
I20260924 20:08:50.230865 9763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:41259 every 8 connection(s)
I20260924 20:08:50.232043 9697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:50.236135 9764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:50.241458 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9697
I20260924 20:08:50.241555 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:50.241892 462 external_mini_cluster.cc:1428] Setting key c0b19fbdfff226b617308263d82e2249
I20260924 20:08:50.242034 9764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe: Bootstrap starting.
I20260924 20:08:50.244907 9764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:50.245806 9764 log.cc:824] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:50.248090 9764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe: No bootstrap required, opened a new log
I20260924 20:08:50.252978 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:50.243359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.0.115.190:41259)
0924 20:08:50.244096 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:50.244116 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:50.244168 (+ 52us) server_negotiation.cc:407] Connection header received
0924 20:08:50.245036 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:50.245062 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:50.245602 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:50.245970 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:50.246405 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.247809 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.249076 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.249769 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.249935 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:50.250746 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:50.250752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:50.250770 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:50.251315 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:50.251389 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:50.251420 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:50.251551 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:50.251633 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 20:08:50.251987 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260924 20:08:50.252825 9764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41259 } }
I20260924 20:08:50.253085 9764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:50.253129 9764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c422607d0494c3aaf8c99dea7cdccfe, State: Initialized, Role: FOLLOWER
I20260924 20:08:50.253654 9764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41259 } }
I20260924 20:08:50.253809 9764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:50.253876 9764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:50.253974 9764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:50.254882 9764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41259 } }
I20260924 20:08:50.255254 9764 leader_election.cc:302] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c422607d0494c3aaf8c99dea7cdccfe; no voters:
I20260924 20:08:50.255553 9764 leader_election.cc:288] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:50.255640 9769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:50.255869 9769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [term 1 LEADER]: Becoming Leader. State: Replica: 2c422607d0494c3aaf8c99dea7cdccfe, State: Running, Role: LEADER
I20260924 20:08:50.256209 9769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41259 } }
I20260924 20:08:50.256752 9764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:50.258072 9770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41259 } } }
I20260924 20:08:50.258219 9770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:50.258478 9771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c422607d0494c3aaf8c99dea7cdccfe. Latest consensus state: current_term: 1 leader_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c422607d0494c3aaf8c99dea7cdccfe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41259 } } }
I20260924 20:08:50.258572 9771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:50.258718 9778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:50.262019 9778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:50.267905 9778 catalog_manager.cc:1347] Generated new cluster ID: 1387f0cf8ddc41bfa11b8ee1e1d3d8da
I20260924 20:08:50.268004 9778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:50.275117 9778 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:50.276155 9778 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:50.285045 9778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c422607d0494c3aaf8c99dea7cdccfe: Generated new TSK 0
I20260924 20:08:50.285797 9778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:08:50.290570 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41259
--builtin_ntp_servers=127.0.115.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 20:08:50.407070 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:50.407333 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:50.407400 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:50.411381 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:50.411432 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:50.411485 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:50.416332 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:50.417515 9788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:50.418953 9788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:50.428843 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:50.428956 9788 server_base.cc:1034] running on GCE node
W20260924 20:08:50.428848 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:50.428862 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:50.429500 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:50.430224 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:50.431401 9788 hybrid_clock.cc:648] HybridClock initialized: now 1790280530431386 us; error 26 us; skew 500 ppm
I20260924 20:08:50.433367 9788 webserver.cc:460] Webserver started at http://127.0.115.129:42783/ using document root <none> and password file <none>
I20260924 20:08:50.433974 9788 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:50.434070 9788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:50.434312 9788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:50.436503 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "0ef4a2a484854b7fa8c76aab61b42968"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "8f77c7802b4b2a1d04936c5c37a88e40"
server_key_iv: "435db4ff1c56520a92fcf50c826f957a"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.437156 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "0ef4a2a484854b7fa8c76aab61b42968"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "8f77c7802b4b2a1d04936c5c37a88e40"
server_key_iv: "435db4ff1c56520a92fcf50c826f957a"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.440869 9788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:08:50.443430 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.444725 9788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:50.444862 9788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "0ef4a2a484854b7fa8c76aab61b42968"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "8f77c7802b4b2a1d04936c5c37a88e40"
server_key_iv: "435db4ff1c56520a92fcf50c826f957a"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.444979 9788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:50.457966 9788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:50.458765 9788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:50.458964 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:50.459636 9788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:50.460666 9788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:50.460738 9788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.460811 9788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:50.460853 9788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.469552 9788 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:35661
I20260924 20:08:50.469637 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:35661 every 8 connection(s)
I20260924 20:08:50.470530 9788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:50.477635 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9788
I20260924 20:08:50.477785 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:50.478083 462 external_mini_cluster.cc:1428] Setting key a55dedaa016100372eb946761d82a46a
I20260924 20:08:50.480348 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41259
--builtin_ntp_servers=127.0.115.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 20:08:50.480505 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:50.472549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:41553 (local address 127.0.115.190:41259)
0924 20:08:50.472702 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:50.472707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:50.473729 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 20:08:50.475221 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:50.475224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:50.475296 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:50.475399 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:50.476357 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.476930 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.478281 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.478587 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.478703 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:50.479999 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:50.480008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:50.480012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:50.480023 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:50.480041 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:50.480049 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:50.480136 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:50.480304 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 20:08:50.480366 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:08:50.481345 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:50.472866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41259 (local address 127.0.115.129:41553)
0924 20:08:50.473534 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:50.473587 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:50.474974 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:50.475591 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:50.475597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:50.475685 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:50.476209 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:50.476223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.477073 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.477077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:50.478107 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:50.478148 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.478793 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.478798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:50.479002 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:50.479701 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:50.479703 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:50.479746 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:50.479814 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:50.480138 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:50.480149 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:50.480157 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:08:50.480292 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 20:08:50.480586 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260924 20:08:50.482707 9916 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41259
I20260924 20:08:50.482969 9916 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:50.483588 9916 heartbeater.cc:511] Master 127.0.115.190:41259 requested a full tablet report, sending...
I20260924 20:08:50.485289 9728 ts_manager.cc:194] Registered new tserver with Master: 0ef4a2a484854b7fa8c76aab61b42968 (127.0.115.129:35661)
I20260924 20:08:50.486464 9728 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.129:41553
W20260924 20:08:50.596874 9919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:50.597143 9919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:50.597180 9919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:50.601300 9919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:50.601359 9919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:50.601414 9919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:50.606189 9919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:50.607293 9919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:50.608759 9919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:50.618686 9919 server_base.cc:1034] running on GCE node
W20260924 20:08:50.618628 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:50.618598 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:50.618657 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:50.619297 9919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:50.620018 9919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:50.621235 9919 hybrid_clock.cc:648] HybridClock initialized: now 1790280530621199 us; error 46 us; skew 500 ppm
I20260924 20:08:50.623080 9919 webserver.cc:460] Webserver started at http://127.0.115.130:42879/ using document root <none> and password file <none>
I20260924 20:08:50.623703 9919 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:50.623767 9919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:50.624078 9919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:50.626367 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "9e2531fa12bf47ceac8f6bb67ad240cb"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "f4b1570a8c12c005d1b418eb1f516bcd"
server_key_iv: "47896ae105217f94f59f521cf711cd78"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.626941 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "9e2531fa12bf47ceac8f6bb67ad240cb"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "f4b1570a8c12c005d1b418eb1f516bcd"
server_key_iv: "47896ae105217f94f59f521cf711cd78"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.630741 9919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 20:08:50.633450 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.634827 9919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:08:50.634967 9919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "9e2531fa12bf47ceac8f6bb67ad240cb"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "f4b1570a8c12c005d1b418eb1f516bcd"
server_key_iv: "47896ae105217f94f59f521cf711cd78"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.635092 9919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:50.648118 9919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:50.648782 9919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:50.648976 9919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:50.649572 9919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:50.650537 9919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:50.650583 9919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.650655 9919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:50.650694 9919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.660554 9919 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:33025
I20260924 20:08:50.660576 10046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:33025 every 8 connection(s)
I20260924 20:08:50.661463 9919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:50.667750 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 9919
I20260924 20:08:50.667881 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:50.668313 462 external_mini_cluster.cc:1428] Setting key de9b7d20a638ea2ffb9e32c1357b41e7
I20260924 20:08:50.670538 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41259
--builtin_ntp_servers=127.0.115.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 20:08:50.671720 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:50.663431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:42001 (local address 127.0.115.190:41259)
0924 20:08:50.663615 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:50.663620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:50.664874 (+ 1254us) server_negotiation.cc:407] Connection header received
0924 20:08:50.666249 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:50.666252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:50.666300 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:50.666380 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:50.667284 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.667922 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.669487 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.669806 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.669904 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:50.671221 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:50.671224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:50.671226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:50.671239 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:50.671257 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:50.671265 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:50.671325 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:50.671512 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 20:08:50.671563 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:08:50.672715 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:50.663929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41259 (local address 127.0.115.130:42001)
0924 20:08:50.664694 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:50.664736 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:50.666033 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:50.666549 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:50.666556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:50.666640 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:50.667141 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:50.667154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.668106 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.668112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:50.669361 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:50.669371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.669980 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.669984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:50.670181 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:50.670898 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:50.670903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:50.670962 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:50.671055 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:50.671353 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:50.671365 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:50.671374 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:08:50.671504 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 20:08:50.671830 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":160,"threads_started":1}
I20260924 20:08:50.673962 10047 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41259
I20260924 20:08:50.674207 10047 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:50.674744 10047 heartbeater.cc:511] Master 127.0.115.190:41259 requested a full tablet report, sending...
I20260924 20:08:50.675819 9728 ts_manager.cc:194] Registered new tserver with Master: 9e2531fa12bf47ceac8f6bb67ad240cb (127.0.115.130:33025)
I20260924 20:08:50.676457 9728 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.130:42001
W20260924 20:08:50.783998 10050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:50.784267 10050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:50.784307 10050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:50.788558 10050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:50.788615 10050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:50.788668 10050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:50.793411 10050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:50.794484 10050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:50.795854 10050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:50.806921 10050 server_base.cc:1034] running on GCE node
W20260924 20:08:50.806916 10058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:50.806912 10055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:50.807189 10056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:50.807629 10050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:50.808351 10050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:50.809559 10050 hybrid_clock.cc:648] HybridClock initialized: now 1790280530809513 us; error 56 us; skew 500 ppm
I20260924 20:08:50.811643 10050 webserver.cc:460] Webserver started at http://127.0.115.131:41919/ using document root <none> and password file <none>
I20260924 20:08:50.812376 10050 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:50.812472 10050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:50.812718 10050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:50.814867 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "de4b002cd79f441aad8f61a87f3f8d88"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "c269472b8184ba43733081e41a76c555"
server_key_iv: "d4bfaf82aeaf5fb623bf9700f7458682"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.815469 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "de4b002cd79f441aad8f61a87f3f8d88"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "c269472b8184ba43733081e41a76c555"
server_key_iv: "d4bfaf82aeaf5fb623bf9700f7458682"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.819326 10050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:08:50.822227 10064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.823535 10050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:50.823719 10050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "de4b002cd79f441aad8f61a87f3f8d88"
format_stamp: "Formatted at 2026-09-24 20:08:50 on dist-test-slave-2pm7"
server_key: "c269472b8184ba43733081e41a76c555"
server_key_iv: "d4bfaf82aeaf5fb623bf9700f7458682"
server_key_version: "encryptionkey@0"
I20260924 20:08:50.823848 10050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:50.846259 10050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:50.847023 10050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:50.847250 10050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:50.847879 10050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:50.848995 10050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:50.849066 10050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.849148 10050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:50.849200 10050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:50.858481 10050 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:45971
I20260924 20:08:50.858500 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:45971 every 8 connection(s)
I20260924 20:08:50.859516 10050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:50.868116 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10050
I20260924 20:08:50.868248 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:50.868608 462 external_mini_cluster.cc:1428] Setting key e8436d01abae9069591aabce305cef7f
I20260924 20:08:50.869697 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:50.861376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:44199 (local address 127.0.115.190:41259)
0924 20:08:50.861524 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:50.861528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:50.862710 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 20:08:50.864101 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:50.864104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:50.864155 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:50.864235 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:50.865214 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.865798 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.867011 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.867245 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.867340 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:50.869047 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:50.869054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:50.869060 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:50.869082 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:50.869134 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:50.869148 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:50.869243 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:50.869433 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 20:08:50.869510 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:08:50.871347 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:50.861806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41259 (local address 127.0.115.131:44199)
0924 20:08:50.862494 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:50.862539 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:50.863817 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:50.864407 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:50.864414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:50.864500 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:50.865031 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:50.865043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.865934 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.865937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:50.866869 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:50.866879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:50.867432 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:50.867435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:50.867657 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:50.868501 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:50.868504 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:50.868571 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:50.868677 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:50.869244 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:50.869256 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:50.869263 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:08:50.869862 (+ 599us) client_negotiation.cc:240] Negotiation successful
0924 20:08:50.870187 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260924 20:08:50.872711 10178 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41259
I20260924 20:08:50.872972 10178 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:50.873528 10178 heartbeater.cc:511] Master 127.0.115.190:41259 requested a full tablet report, sending...
I20260924 20:08:50.874612 9728 ts_manager.cc:194] Registered new tserver with Master: de4b002cd79f441aad8f61a87f3f8d88 (127.0.115.131:45971)
I20260924 20:08:50.875229 9728 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.131:44199
I20260924 20:08:50.883214 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:08:50.886178 9767 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:50.885171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.0.115.190:41259)
0924 20:08:50.885331 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:50.885335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:50.885502 (+ 167us) server_negotiation.cc:407] Connection header received
0924 20:08:50.885667 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:50.885670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:50.885723 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:50.885805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:50.886030 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50800: BlockingRecv error: recv got EOF from 127.0.0.1:50800 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:08:50.886328 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9788
I20260924 20:08:50.892421 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9919
I20260924 20:08:50.898277 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10050
I20260924 20:08:50.903645 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 9697
2026-09-24T20:08:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (918 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T20:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:50Z Disabled control of system clock
I20260924 20:08:50.933919 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33393
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:33393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 20:08:51.043735 10189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:51.044094 10189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:51.044137 10189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:51.048043 10189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:51.048094 10189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:51.048108 10189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:51.048125 10189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:51.048139 10189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:51.052877 10189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:33393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:51.053968 10189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:51.055317 10189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:51.063977 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.063959 10195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.064043 10194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:51.064230 10189 server_base.cc:1034] running on GCE node
I20260924 20:08:51.064929 10189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:51.066056 10189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:51.067246 10189 hybrid_clock.cc:648] HybridClock initialized: now 1790280531067231 us; error 28 us; skew 500 ppm
I20260924 20:08:51.069289 10189 webserver.cc:460] Webserver started at http://127.0.115.190:39763/ using document root <none> and password file <none>
I20260924 20:08:51.069916 10189 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:51.069988 10189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:51.070235 10189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:51.072420 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "8cac964585b544928f58a6197a5731fe"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "061aaf0ec528d7824b1f377d68da9171"
server_key_iv: "a1553aed8b03b9d4819cbadee02ce99a"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.072985 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "8cac964585b544928f58a6197a5731fe"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "061aaf0ec528d7824b1f377d68da9171"
server_key_iv: "a1553aed8b03b9d4819cbadee02ce99a"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.076767 10189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:51.079304 10203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.080688 10189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:08:51.080835 10189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "8cac964585b544928f58a6197a5731fe"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "061aaf0ec528d7824b1f377d68da9171"
server_key_iv: "a1553aed8b03b9d4819cbadee02ce99a"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.080946 10189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:51.085664 10189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:51.086316 10189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:51.086498 10189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:51.093653 10189 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:33393
I20260924 20:08:51.093662 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:33393 every 8 connection(s)
I20260924 20:08:51.094664 10189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:51.098032 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:51.099292 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10189
I20260924 20:08:51.099385 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:51.099720 462 external_mini_cluster.cc:1428] Setting key 2c308524ef02fda861351d5742f0bb5b
I20260924 20:08:51.104482 10256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe: Bootstrap starting.
I20260924 20:08:51.105593 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:51.101342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.115.190:33393)
0924 20:08:51.102043 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:51.102059 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:51.102095 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:08:51.102727 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:51.102743 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:51.103096 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:51.103450 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:51.104207 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:51.104220 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:51.104232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:51.104242 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:51.104296 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:51.104311 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:51.104466 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:51.104535 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 20:08:51.104912 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260924 20:08:51.107010 10256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:51.107839 10256 log.cc:824] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:51.109863 10256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe: No bootstrap required, opened a new log
I20260924 20:08:51.114172 10256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cac964585b544928f58a6197a5731fe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33393 } }
I20260924 20:08:51.114431 10256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:51.114506 10256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cac964585b544928f58a6197a5731fe, State: Initialized, Role: FOLLOWER
I20260924 20:08:51.114931 10256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cac964585b544928f58a6197a5731fe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33393 } }
I20260924 20:08:51.115085 10256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:51.115173 10256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:51.115276 10256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:51.116277 10256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cac964585b544928f58a6197a5731fe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33393 } }
I20260924 20:08:51.116621 10256 leader_election.cc:302] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cac964585b544928f58a6197a5731fe; no voters:
I20260924 20:08:51.116938 10256 leader_election.cc:288] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:51.117100 10261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:51.117334 10261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [term 1 LEADER]: Becoming Leader. State: Replica: 8cac964585b544928f58a6197a5731fe, State: Running, Role: LEADER
I20260924 20:08:51.117635 10261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cac964585b544928f58a6197a5731fe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33393 } }
I20260924 20:08:51.118240 10256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:51.119313 10263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cac964585b544928f58a6197a5731fe. Latest consensus state: current_term: 1 leader_uuid: "8cac964585b544928f58a6197a5731fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cac964585b544928f58a6197a5731fe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33393 } } }
I20260924 20:08:51.119442 10263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:51.119834 10262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cac964585b544928f58a6197a5731fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cac964585b544928f58a6197a5731fe" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33393 } } }
I20260924 20:08:51.119895 10270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:51.119959 10262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:51.122663 10270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:51.128464 10270 catalog_manager.cc:1347] Generated new cluster ID: 42da87247c444a7f97c1f99d0fbcdef9
I20260924 20:08:51.128564 10270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:51.143481 10270 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:51.143837 10270 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:51.150336 10270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cac964585b544928f58a6197a5731fe: Generated new TSK 0
I20260924 20:08:51.151022 10270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:08:51.163262 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33393
--builtin_ntp_servers=127.0.115.148:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 20:08:51.275189 10280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:51.275489 10280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:51.275563 10280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:51.279549 10280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:51.279629 10280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:51.279722 10280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:51.284546 10280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:51.285706 10280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:51.287073 10280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:08:51.297120 10280 server_base.cc:1034] running on GCE node
W20260924 20:08:51.297034 10285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.297034 10286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.297027 10288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:51.297780 10280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:51.298431 10280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:51.299672 10280 hybrid_clock.cc:648] HybridClock initialized: now 1790280531299653 us; error 26 us; skew 500 ppm
I20260924 20:08:51.306934 10280 webserver.cc:460] Webserver started at http://127.0.115.129:35941/ using document root <none> and password file <none>
I20260924 20:08:51.307631 10280 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:51.307760 10280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:51.308110 10280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:51.310468 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "fdadb5bf77654ada95d6cef6274f6c64"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "a7865677c5f661b63ae39d16237df802"
server_key_iv: "d64eece55004fda11a6996d812e45f99"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.311089 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "fdadb5bf77654ada95d6cef6274f6c64"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "a7865677c5f661b63ae39d16237df802"
server_key_iv: "d64eece55004fda11a6996d812e45f99"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.315135 10280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:08:51.317982 10294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.319424 10280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:51.319602 10280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "fdadb5bf77654ada95d6cef6274f6c64"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "a7865677c5f661b63ae39d16237df802"
server_key_iv: "d64eece55004fda11a6996d812e45f99"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.319746 10280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:51.324894 10280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:51.325534 10280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:51.325742 10280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:51.326359 10280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:51.327333 10280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:51.327453 10280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.327569 10280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:51.327607 10280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.336841 10280 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:36113
I20260924 20:08:51.336887 10407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:36113 every 8 connection(s)
I20260924 20:08:51.337855 10280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:51.339986 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10280
I20260924 20:08:51.340106 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:51.340399 462 external_mini_cluster.cc:1428] Setting key 8dac7c5defdc4b9c10c9b73c0957d228
I20260924 20:08:51.342650 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33393
--builtin_ntp_servers=127.0.115.148:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 20:08:51.345201 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:51.339879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:45981 (local address 127.0.115.190:33393)
0924 20:08:51.340110 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:51.340115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:51.341461 (+ 1346us) server_negotiation.cc:407] Connection header received
0924 20:08:51.343249 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:51.343252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:51.343292 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:51.343377 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:51.344686 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:51.344693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:51.344697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:51.344714 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:51.344738 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:51.344751 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:51.344858 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:51.344989 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 20:08:51.345039 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 20:08:51.346050 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:51.340401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33393 (local address 127.0.115.129:45981)
0924 20:08:51.341314 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:51.341362 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:51.342966 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:51.343558 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:51.343569 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:51.343704 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:51.344401 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:51.344403 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:51.344444 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:51.344517 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:51.344845 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:51.344851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:51.344856 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:08:51.344989 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 20:08:51.345192 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":234,"threads_started":1}
I20260924 20:08:51.347465 10408 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33393
I20260924 20:08:51.347766 10408 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:51.348259 10408 heartbeater.cc:511] Master 127.0.115.190:33393 requested a full tablet report, sending...
I20260924 20:08:51.350055 10220 ts_manager.cc:194] Registered new tserver with Master: fdadb5bf77654ada95d6cef6274f6c64 (127.0.115.129:36113)
W20260924 20:08:51.476919 10411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:51.477173 10411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:51.477212 10411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:51.481176 10411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:51.481230 10411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:51.481284 10411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:51.486227 10411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:51.487576 10411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:51.489506 10411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:51.499864 10416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.499866 10417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.499887 10419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:51.500272 10411 server_base.cc:1034] running on GCE node
I20260924 20:08:51.500663 10411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:51.501241 10411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:51.502405 10411 hybrid_clock.cc:648] HybridClock initialized: now 1790280531502353 us; error 64 us; skew 500 ppm
I20260924 20:08:51.504333 10411 webserver.cc:460] Webserver started at http://127.0.115.130:38721/ using document root <none> and password file <none>
I20260924 20:08:51.504954 10411 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:51.505016 10411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:51.505232 10411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:51.507334 10411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "9fae8a10de8c4d93963b6c6653c72f1c"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "4981a700861aa6cc403d2a3ad0076ad2"
server_key_iv: "8da8f9ca325370786c97c212ae418bca"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.507900 10411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "9fae8a10de8c4d93963b6c6653c72f1c"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "4981a700861aa6cc403d2a3ad0076ad2"
server_key_iv: "8da8f9ca325370786c97c212ae418bca"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.511606 10411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:08:51.514364 10425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.515715 10411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:08:51.515884 10411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "9fae8a10de8c4d93963b6c6653c72f1c"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "4981a700861aa6cc403d2a3ad0076ad2"
server_key_iv: "8da8f9ca325370786c97c212ae418bca"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.516083 10411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:51.521520 10411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:51.522192 10411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:51.522392 10411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:51.522992 10411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:51.524077 10411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:51.524168 10411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.524232 10411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:51.524274 10411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.534106 10411 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:36301
I20260924 20:08:51.534119 10538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:36301 every 8 connection(s)
I20260924 20:08:51.535054 10411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:51.540858 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10411
I20260924 20:08:51.540968 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:51.541380 462 external_mini_cluster.cc:1428] Setting key 63ab8d2aac308ce66a170010fa2d40f8
I20260924 20:08:51.541775 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:51.537060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35395 (local address 127.0.115.190:33393)
0924 20:08:51.537227 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:51.537231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:51.538300 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 20:08:51.539702 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:51.539707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:51.539762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:51.539879 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:51.541252 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:51.541257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:51.541260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:51.541275 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:51.541300 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:51.541313 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:51.541419 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:51.541590 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 20:08:51.541640 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:08:51.542979 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:51.537449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33393 (local address 127.0.115.130:35395)
0924 20:08:51.538145 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:51.538185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:51.539425 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:51.540092 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:51.540099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:51.540192 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:51.540860 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:51.540862 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:51.540933 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:51.541063 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:51.541409 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:51.541416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:51.541421 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:08:51.541620 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 20:08:51.541891 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260924 20:08:51.543772 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33393
--builtin_ntp_servers=127.0.115.148:36671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 20:08:51.544303 10539 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33393
I20260924 20:08:51.544560 10539 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:51.545029 10539 heartbeater.cc:511] Master 127.0.115.190:33393 requested a full tablet report, sending...
I20260924 20:08:51.546195 10220 ts_manager.cc:194] Registered new tserver with Master: 9fae8a10de8c4d93963b6c6653c72f1c (127.0.115.130:36301)
W20260924 20:08:51.657060 10542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:51.657313 10542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:51.657354 10542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:51.661484 10542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:51.661531 10542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:51.661586 10542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:51.666253 10542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:51.667248 10542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:51.668656 10542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:51.678081 10547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.678157 10548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:51.678092 10550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:51.678413 10542 server_base.cc:1034] running on GCE node
I20260924 20:08:51.678789 10542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:51.679406 10542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:51.680583 10542 hybrid_clock.cc:648] HybridClock initialized: now 1790280531680561 us; error 38 us; skew 500 ppm
I20260924 20:08:51.682340 10542 webserver.cc:460] Webserver started at http://127.0.115.131:44395/ using document root <none> and password file <none>
I20260924 20:08:51.682909 10542 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:51.682996 10542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:51.683226 10542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:51.685316 10542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "9a40baa4627e429f80abb647ba350c68"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "ddf399bc1f23b551bc68241f5321c5ad"
server_key_iv: "c9c9fba307fd077811b963a4f349cdce"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.685907 10542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "9a40baa4627e429f80abb647ba350c68"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "ddf399bc1f23b551bc68241f5321c5ad"
server_key_iv: "c9c9fba307fd077811b963a4f349cdce"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.689491 10542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:08:51.691836 10556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.693046 10542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:51.693168 10542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "9a40baa4627e429f80abb647ba350c68"
format_stamp: "Formatted at 2026-09-24 20:08:51 on dist-test-slave-2pm7"
server_key: "ddf399bc1f23b551bc68241f5321c5ad"
server_key_iv: "c9c9fba307fd077811b963a4f349cdce"
server_key_version: "encryptionkey@0"
I20260924 20:08:51.693274 10542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:51.697801 10542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:51.698333 10542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:51.698459 10542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:51.698915 10542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:51.699720 10542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:51.699764 10542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.699801 10542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:51.699817 10542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:51.708428 10542 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:39699
I20260924 20:08:51.708451 10669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:39699 every 8 connection(s)
I20260924 20:08:51.709345 10542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:51.710806 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10542
I20260924 20:08:51.710922 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:51.711246 462 external_mini_cluster.cc:1428] Setting key f7d9b39635099f7b96420e35790bef87
I20260924 20:08:51.716063 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:51.711329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:41795 (local address 127.0.115.190:33393)
0924 20:08:51.711484 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:51.711497 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:51.712833 (+ 1336us) server_negotiation.cc:407] Connection header received
0924 20:08:51.714382 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:51.714385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:51.714421 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:51.714491 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:51.715639 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:51.715642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:51.715645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:08:51.715655 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:51.715673 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:08:51.715681 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:51.715748 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:51.715858 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 20:08:51.715894 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:08:51.716944 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:51.711851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33393 (local address 127.0.115.131:41795)
0924 20:08:51.712678 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:51.712750 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:51.714172 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:51.714692 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:51.714699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:51.714787 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:51.715399 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:08:51.715400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:51.715445 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:08:51.715512 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:51.715747 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:51.715752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:51.715755 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:08:51.715876 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:08:51.716128 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":139,"threads_started":1}
I20260924 20:08:51.718029 10670 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33393
I20260924 20:08:51.718289 10670 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:51.718770 10670 heartbeater.cc:511] Master 127.0.115.190:33393 requested a full tablet report, sending...
I20260924 20:08:51.720014 10220 ts_manager.cc:194] Registered new tserver with Master: 9a40baa4627e429f80abb647ba350c68 (127.0.115.131:39699)
I20260924 20:08:51.725378 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:08:51.728370 10259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:08:51.727332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.0.115.190:33393)
0924 20:08:51.727496 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:51.727500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:51.727687 (+ 187us) server_negotiation.cc:407] Connection header received
0924 20:08:51.727849 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:51.727852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:51.727888 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:51.728009 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:51.728243 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34476: BlockingRecv error: recv got EOF from 127.0.0.1:34476 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:08:51.728822 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10280
I20260924 20:08:51.734887 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10411
I20260924 20:08:51.740322 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10542
I20260924 20:08:51.745605 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10189
2026-09-24T20:08:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (840 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:51 dist-test-slave-2pm7 krb5kdc[10680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:51 dist-test-slave-2pm7 krb5kdc[10680](info): set up 2 sockets
Sep 24 20:08:51 dist-test-slave-2pm7 krb5kdc[10680](info): commencing operation
krb5kdc: starting...
W20260924 20:08:52.841120 462 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:52 dist-test-slave-2pm7 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:08:52.968096 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34257
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:53.078218 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:53.078473 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:53.078514 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:53.082369 10693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:08:53.082417 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:53.082434 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:53.082450 10693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:08:53.082465 10693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:08:53.087509 10693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:34257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:53.088481 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:53.089432 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:53.097769 10701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:53.097822 10699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:53.097892 10693 server_base.cc:1034] running on GCE node
W20260924 20:08:53.097769 10698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:53.098541 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:53.099581 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:53.100739 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790280533100709 us; error 39 us; skew 500 ppm
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:53.103493 10693 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:08:53.104373 10693 webserver.cc:460] Webserver started at http://127.0.115.190:40765/ using document root <none> and password file <none>
I20260924 20:08:53.104897 10693 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:53.104952 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:53.105115 10693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:53.107055 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "d4a533515c8140bbaf681207fcd8884d"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "b525db0bb6ec16c4ec578f279b35d33a"
server_key_iv: "3ae7e0e85346ca4efe5d161b2d1e93cc"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.107549 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "d4a533515c8140bbaf681207fcd8884d"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "b525db0bb6ec16c4ec578f279b35d33a"
server_key_iv: "3ae7e0e85346ca4efe5d161b2d1e93cc"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.111447 10693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:08:53.113915 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.115312 10693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:53.115463 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "d4a533515c8140bbaf681207fcd8884d"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "b525db0bb6ec16c4ec578f279b35d33a"
server_key_iv: "3ae7e0e85346ca4efe5d161b2d1e93cc"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.115577 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:53.138132 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:53.141537 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:53.141757 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:53.148912 10693 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:34257
I20260924 20:08:53.149209 10760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:34257 every 8 connection(s)
I20260924 20:08:53.149891 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:08:53.152791 10761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:53.154508 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10693
I20260924 20:08:53.154633 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:08:53.154930 462 external_mini_cluster.cc:1428] Setting key 9f0ff1219cc63ceec67da50db11ff910
I20260924 20:08:53.159422 10761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d: Bootstrap starting.
I20260924 20:08:53.162164 10761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:53.163173 10761 log.cc:824] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:53.165756 10761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d: No bootstrap required, opened a new log
I20260924 20:08:53.170096 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.156360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.0.115.190:34257)
0924 20:08:53.156777 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.156794 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.156829 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:08:53.157453 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.157470 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.157809 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.158081 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:53.159001 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.160104 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.161256 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.161727 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.161846 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.164835 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:53.164842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:53.164859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:53.164891 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:53.166882 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.167329 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.167337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.167345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.167482 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.167772 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.167775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.167777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.168405 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:53.168650 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.168804 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.169120 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":127,"threads_started":1}
I20260924 20:08:53.170590 10761 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a533515c8140bbaf681207fcd8884d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34257 } }
I20260924 20:08:53.170861 10761 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:53.170933 10761 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a533515c8140bbaf681207fcd8884d, State: Initialized, Role: FOLLOWER
I20260924 20:08:53.171398 10761 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a533515c8140bbaf681207fcd8884d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34257 } }
I20260924 20:08:53.171525 10761 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:08:53.171602 10761 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:08:53.171700 10761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:53.172849 10761 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a533515c8140bbaf681207fcd8884d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34257 } }
I20260924 20:08:53.173194 10761 leader_election.cc:302] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a533515c8140bbaf681207fcd8884d; no voters:
I20260924 20:08:53.173496 10761 leader_election.cc:288] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:08:53.173677 10766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:53.173909 10766 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [term 1 LEADER]: Becoming Leader. State: Replica: d4a533515c8140bbaf681207fcd8884d, State: Running, Role: LEADER
I20260924 20:08:53.174214 10766 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a533515c8140bbaf681207fcd8884d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34257 } }
I20260924 20:08:53.174803 10761 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:08:53.176138 10767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a533515c8140bbaf681207fcd8884d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a533515c8140bbaf681207fcd8884d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34257 } } }
I20260924 20:08:53.176118 10768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a533515c8140bbaf681207fcd8884d. Latest consensus state: current_term: 1 leader_uuid: "d4a533515c8140bbaf681207fcd8884d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a533515c8140bbaf681207fcd8884d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34257 } } }
I20260924 20:08:53.176268 10768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [sys.catalog]: This master's current role is: LEADER
I20260924 20:08:53.176268 10767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d [sys.catalog]: This master's current role is: LEADER
W20260924 20:08:53.179755 10781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:08:53.179853 10781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:08:53.179924 10776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:08:53.180644 10776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:08:53.186347 10776 catalog_manager.cc:1347] Generated new cluster ID: 72655ae74a214c9ca6d23c8c42ef20e3
I20260924 20:08:53.186414 10776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:08:53.198679 10776 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:08:53.200048 10776 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:08:53.209746 10776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d: Generated new TSK 0
I20260924 20:08:53.210495 10776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:08:53.245131 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34257
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:53.360391 10787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:53.360654 10787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:53.360704 10787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:53.364672 10787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:53.364729 10787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:53.364785 10787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:08:53.369994 10787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:53.371102 10787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:53.372184 10787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:53.382045 10792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:53.382064 10795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:53.382043 10793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:53.382159 10787 server_base.cc:1034] running on GCE node
I20260924 20:08:53.382712 10787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:53.383358 10787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:53.384563 10787 hybrid_clock.cc:648] HybridClock initialized: now 1790280533384531 us; error 51 us; skew 500 ppm
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:53.387480 10787 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:08:53.388368 10787 webserver.cc:460] Webserver started at http://127.0.115.129:33815/ using document root <none> and password file <none>
I20260924 20:08:53.388913 10787 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:53.389001 10787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:53.389240 10787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:53.391228 10787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "f757e42d4c494588ad00add362c488ed"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "75ddfd558e26d04eff5c2ea33893133f"
server_key_iv: "7041a150ef7d447940209557da2af00d"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.391784 10787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "f757e42d4c494588ad00add362c488ed"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "75ddfd558e26d04eff5c2ea33893133f"
server_key_iv: "7041a150ef7d447940209557da2af00d"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.395617 10787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:08:53.398288 10802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.399559 10787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:08:53.399694 10787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "f757e42d4c494588ad00add362c488ed"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "75ddfd558e26d04eff5c2ea33893133f"
server_key_iv: "7041a150ef7d447940209557da2af00d"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.399807 10787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:53.411231 10787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:53.414165 10787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:53.414381 10787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:53.414981 10787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:53.415827 10787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:53.415897 10787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.415982 10787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:53.416025 10787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.425082 10787 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:38425
I20260924 20:08:53.425143 10915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:38425 every 8 connection(s)
I20260924 20:08:53.426023 10787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:08:53.431560 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10787
I20260924 20:08:53.431679 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:08:53.432044 462 external_mini_cluster.cc:1428] Setting key 5ff7d77fa40cfa64d576048912b93915
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:53.440814 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.427923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:35771 (local address 127.0.115.190:34257)
0924 20:08:53.428086 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.428091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.429419 (+ 1328us) server_negotiation.cc:407] Connection header received
0924 20:08:53.430297 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.430300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.430345 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.430428 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:53.431715 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.432402 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.433917 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.434316 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.434439 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.437206 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:53.437209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:53.437211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:53.437234 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:53.438683 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.439197 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.439205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.439206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.439278 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.439696 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.439699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.439701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.439876 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:53.439999 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.440500 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.440640 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:08:53.441659 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.428399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:34257 (local address 127.0.115.129:35771)
0924 20:08:53.429269 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.429312 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.430104 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.430582 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.430635 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.430970 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:53.431540 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.431563 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.432529 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.432532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.433791 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.433801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.434500 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.434504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.434665 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.435308 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:53.435334 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:53.437029 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:53.438801 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:53.438808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:53.438823 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:53.439080 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:53.439366 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:53.439369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:53.439372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:53.439526 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:53.439999 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:53.440006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:53.440260 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.440448 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.440770 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":532,"threads_started":1}
I20260924 20:08:53.442950 10916 heartbeater.cc:347] Connected to a master server at 127.0.115.190:34257
I20260924 20:08:53.443168 10916 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:53.443758 10916 heartbeater.cc:511] Master 127.0.115.190:34257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:08:53.445415 10725 ts_manager.cc:194] Registered new tserver with Master: f757e42d4c494588ad00add362c488ed (127.0.115.129:38425)
I20260924 20:08:53.446758 10725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:35771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:08:53.459151 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34257
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:53.571414 10921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:53.571674 10921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:53.571717 10921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:53.576004 10921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:53.576064 10921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:53.576159 10921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:08:53.581470 10921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:53.582580 10921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:53.583623 10921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:53.593067 10929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:53.593067 10927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:53.593062 10926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:53.593432 10921 server_base.cc:1034] running on GCE node
I20260924 20:08:53.593809 10921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:53.594458 10921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:53.595619 10921 hybrid_clock.cc:648] HybridClock initialized: now 1790280533595600 us; error 28 us; skew 500 ppm
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:53.598321 10921 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:08:53.599145 10921 webserver.cc:460] Webserver started at http://127.0.115.130:42877/ using document root <none> and password file <none>
I20260924 20:08:53.599697 10921 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:53.599785 10921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:53.600072 10921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:53.602056 10921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "1659466ed922489d80c8d081346d6b97"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "0e4edba92bcb75e156af8bc6e6a4cd20"
server_key_iv: "040ae4b5e2d964451fb18bd412769e51"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.602698 10921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "1659466ed922489d80c8d081346d6b97"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "0e4edba92bcb75e156af8bc6e6a4cd20"
server_key_iv: "040ae4b5e2d964451fb18bd412769e51"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.606508 10921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:08:53.608944 10936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.610206 10921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:53.610363 10921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "1659466ed922489d80c8d081346d6b97"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "0e4edba92bcb75e156af8bc6e6a4cd20"
server_key_iv: "040ae4b5e2d964451fb18bd412769e51"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.610479 10921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:53.627168 10921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:53.630210 10921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:53.630419 10921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:53.631026 10921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:53.631888 10921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:53.631981 10921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.632081 10921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:53.632131 10921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.641166 10921 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:39879
I20260924 20:08:53.641185 11049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:39879 every 8 connection(s)
I20260924 20:08:53.642100 10921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:08:53.645591 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 10921
I20260924 20:08:53.645748 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:08:53.646064 462 external_mini_cluster.cc:1428] Setting key 2464f18301e15fcb7c85a1eccc8ee70a
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:08:53.656765 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.643902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35391 (local address 127.0.115.190:34257)
0924 20:08:53.644077 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.644082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.645160 (+ 1078us) server_negotiation.cc:407] Connection header received
0924 20:08:53.646303 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.646307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.646369 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.646459 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:53.647919 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.648522 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.650011 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.650406 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.650540 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.652971 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:53.652977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:53.652982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:53.653018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:53.654696 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.655200 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.655206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.655210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.655305 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.655644 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.655650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.655654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.655908 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:53.656031 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.656525 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.656641 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 20:08:53.657557 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.644322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:34257 (local address 127.0.115.130:35391)
0924 20:08:53.644992 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.645035 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.646039 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.646672 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.646681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.647092 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:53.647756 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.647778 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.648681 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.648686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.649882 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.649892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.650563 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.650565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.650702 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.651307 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:53.651323 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:53.652792 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:53.654827 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:53.654832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:53.654844 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:53.655094 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:53.655411 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:53.655413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:53.655414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:53.655539 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:53.656033 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:53.656038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:53.656274 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.656432 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.656686 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:08:53.658902 11050 heartbeater.cc:347] Connected to a master server at 127.0.115.190:34257
I20260924 20:08:53.659174 11050 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:53.659811 11050 heartbeater.cc:511] Master 127.0.115.190:34257 requested a full tablet report, sending...
I20260924 20:08:53.661134 10725 ts_manager.cc:194] Registered new tserver with Master: 1659466ed922489d80c8d081346d6b97 (127.0.115.130:39879)
I20260924 20:08:53.662005 10725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:35391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:08:53.675135 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34257
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:08:53.785753 11055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:08:53.786047 11055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:08:53.786121 11055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:08:53.790328 11055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:08:53.790412 11055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:08:53.790522 11055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:08:53.795802 11055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:08:53.796963 11055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:08:53.798058 11055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:08:53.808346 11061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:53.808369 11063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:08:53.808346 11060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:08:53.808671 11055 server_base.cc:1034] running on GCE node
I20260924 20:08:53.809082 11055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:08:53.809768 11055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:08:53.810950 11055 hybrid_clock.cc:648] HybridClock initialized: now 1790280533810912 us; error 48 us; skew 500 ppm
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:08:53.817055 11055 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:08:53.817894 11055 webserver.cc:460] Webserver started at http://127.0.115.131:37333/ using document root <none> and password file <none>
I20260924 20:08:53.818481 11055 fs_manager.cc:359] Metadata directory not provided
I20260924 20:08:53.818547 11055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:08:53.818725 11055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:08:53.820922 11055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "1ba08890fee24894b7b5ac64150c55c1"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "a42bbeae6a813e44cfc0176f1282b5f8"
server_key_iv: "721b0cc3171372da7d6e316dd6f8e244"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.821430 11055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "1ba08890fee24894b7b5ac64150c55c1"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "a42bbeae6a813e44cfc0176f1282b5f8"
server_key_iv: "721b0cc3171372da7d6e316dd6f8e244"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.825085 11055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 20:08:53.827342 11070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.828590 11055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:08:53.828718 11055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "1ba08890fee24894b7b5ac64150c55c1"
format_stamp: "Formatted at 2026-09-24 20:08:53 on dist-test-slave-2pm7"
server_key: "a42bbeae6a813e44cfc0176f1282b5f8"
server_key_iv: "721b0cc3171372da7d6e316dd6f8e244"
server_key_version: "encryptionkey@0"
I20260924 20:08:53.828804 11055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:08:53.841233 11055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:08:53.844460 11055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:08:53.844609 11055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:08:53.845145 11055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:08:53.845927 11055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:08:53.845973 11055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.846011 11055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:08:53.846031 11055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:08:53.854597 11055 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44373
I20260924 20:08:53.854624 11183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44373 every 8 connection(s)
I20260924 20:08:53.855487 11055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:08:53.861670 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 11055
I20260924 20:08:53.861809 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:08:53.862121 462 external_mini_cluster.cc:1428] Setting key 8e01948440ab146ee5ea3d4538a89fd2
Sep 24 20:08:53 dist-test-slave-2pm7 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:08:53.869643 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.857459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:36983 (local address 127.0.115.190:34257)
0924 20:08:53.857623 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.857628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.858736 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 20:08:53.859586 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.859590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.859638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.859716 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:53.860937 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.861505 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.863085 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.863380 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.863498 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.866028 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:53.866031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:53.866033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:53.866060 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:53.867385 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.867937 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.868005 (+ 68us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.868013 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.868114 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.868471 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.868474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.868475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.868732 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:53.868824 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.869329 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.869446 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:08:53.870421 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.857908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:34257 (local address 127.0.115.131:36983)
0924 20:08:53.858551 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.858590 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.859373 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.859876 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.859883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.860285 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:08:53.860802 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.860816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.861674 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.861678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.862962 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.862974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.863548 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.863552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.863745 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.864397 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:08:53.864419 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:08:53.865863 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:08:53.867495 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:53.867502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:53.867527 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:53.867814 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:53.868211 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:08:53.868214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:08:53.868215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:08:53.868352 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:08:53.868835 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:08:53.868840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:08:53.869088 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.869256 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.869519 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":112,"threads_started":1}
I20260924 20:08:53.871692 11184 heartbeater.cc:347] Connected to a master server at 127.0.115.190:34257
I20260924 20:08:53.871981 11184 heartbeater.cc:464] Registering TS with master...
I20260924 20:08:53.872502 11184 heartbeater.cc:511] Master 127.0.115.190:34257 requested a full tablet report, sending...
I20260924 20:08:53.873621 10725 ts_manager.cc:194] Registered new tserver with Master: 1ba08890fee24894b7b5ac64150c55c1 (127.0.115.131:44373)
I20260924 20:08:53.874178 10725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:36983
I20260924 20:08:53.876215 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:08:53.886385 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.878272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.115.190:34257)
0924 20:08:53.878405 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.878408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.878463 (+ 55us) server_negotiation.cc:407] Connection header received
0924 20:08:53.878581 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.878583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.878620 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.878686 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:08:53.879340 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.879841 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.880697 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.880943 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.881067 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.882766 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:08:53.882773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:08:53.882776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:08:53.882824 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:08:53.884630 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.885059 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.885062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.885064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.885148 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:08:53.885474 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:08:53.885478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:08:53.885479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:08:53.885716 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:08:53.885858 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.886097 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.886225 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:08:53.889405 10725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:08:53.892531 10725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:08:53.908453 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.901800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:39879 (local address 127.0.0.1:54184)
0924 20:08:53.902206 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.902265 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.902422 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.904300 (+ 1878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.904303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.904324 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:53.904445 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.904451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.905763 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.905767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.906945 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.906952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.907787 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.907790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.908043 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.908081 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.908167 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.908286 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":199,"thread_start_us":126,"threads_started":1}
I20260924 20:08:53.908479 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.902911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.115.131:44373)
0924 20:08:53.903298 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.903310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.903332 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:08:53.903409 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.903415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.903631 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.904158 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:53.904346 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.905160 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.906754 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.907618 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.907732 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.907890 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.908250 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.908339 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":190,"threads_started":1}
I20260924 20:08:53.908479 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.901980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.0.115.130:39879)
0924 20:08:53.903762 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.903774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.903799 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:08:53.903879 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.903884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.904150 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.904320 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:53.904542 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.905627 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.907061 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.907632 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.907926 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.908104 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.908221 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.908324 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1670,"thread_start_us":97,"threads_started":1}
I20260924 20:08:53.909178 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.902692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.0.115.129:38425)
0924 20:08:53.903253 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.903307 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.903332 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:08:53.903410 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.903415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.903631 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.903793 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:53.904216 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.905510 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.907169 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.908025 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.908132 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.908276 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.908932 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.909034 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":147,"threads_started":1}
I20260924 20:08:53.909200 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.902617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44373 (local address 127.0.0.1:51778)
0924 20:08:53.902868 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.902882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.902968 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.903815 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.903821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.903846 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:53.904003 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.904011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.905294 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.905299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.906636 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.906648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.907763 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.907769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.907960 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.908006 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.908126 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.909040 (+ 914us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":804,"thread_start_us":89,"threads_started":1}
I20260924 20:08:53.909773 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.902180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:38425 (local address 127.0.0.1:56084)
0924 20:08:53.902456 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.902470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.902573 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.903979 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.903982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.903996 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:53.904111 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.904118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.905694 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.905701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.907025 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.907036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.908334 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.908338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.908432 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.908452 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.908523 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.908606 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":71,"thread_start_us":92,"threads_started":1}
I20260924 20:08:53.912016 11118 tablet_service.cc:1467] Processing CreateTablet for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c (DEFAULT_TABLE table=test-table [id=62d2bbfd14d04ed1a32ad06e4e3cb1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:53.912143 10984 tablet_service.cc:1467] Processing CreateTablet for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c (DEFAULT_TABLE table=test-table [id=62d2bbfd14d04ed1a32ad06e4e3cb1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:53.912480 10850 tablet_service.cc:1467] Processing CreateTablet for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c (DEFAULT_TABLE table=test-table [id=62d2bbfd14d04ed1a32ad06e4e3cb1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:08:53.913192 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:53.913194 11118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:53.913390 10850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:08:53.920104 11201 tablet_bootstrap.cc:492] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Bootstrap starting.
I20260924 20:08:53.921499 11202 tablet_bootstrap.cc:492] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed: Bootstrap starting.
I20260924 20:08:53.921499 11203 tablet_bootstrap.cc:492] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Bootstrap starting.
I20260924 20:08:53.922863 11201 tablet_bootstrap.cc:654] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:53.923662 11201 log.cc:824] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:53.923769 11202 tablet_bootstrap.cc:654] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:53.924281 11203 tablet_bootstrap.cc:654] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Neither blocks nor log segments found. Creating new log.
I20260924 20:08:53.925120 11203 log.cc:824] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:53.925117 11202 log.cc:824] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed: Log is configured to *not* fsync() on all Append() calls
I20260924 20:08:53.926443 11201 tablet_bootstrap.cc:492] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: No bootstrap required, opened a new log
I20260924 20:08:53.926673 11201 ts_tablet_manager.cc:1397] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:08:53.927248 11202 tablet_bootstrap.cc:492] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed: No bootstrap required, opened a new log
I20260924 20:08:53.927436 11202 ts_tablet_manager.cc:1397] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:08:53.927520 11203 tablet_bootstrap.cc:492] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: No bootstrap required, opened a new log
I20260924 20:08:53.927764 11203 ts_tablet_manager.cc:1397] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:08:53.930153 11201 raft_consensus.cc:348] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.930420 11201 raft_consensus.cc:374] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:53.930474 11201 raft_consensus.cc:729] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba08890fee24894b7b5ac64150c55c1, State: Initialized, Role: FOLLOWER
I20260924 20:08:53.930966 11201 consensus_queue.cc:260] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.931393 11203 raft_consensus.cc:348] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.931363 11202 raft_consensus.cc:348] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.931636 11202 raft_consensus.cc:374] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:53.931638 11203 raft_consensus.cc:374] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:08:53.931689 11203 raft_consensus.cc:729] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1659466ed922489d80c8d081346d6b97, State: Initialized, Role: FOLLOWER
I20260924 20:08:53.931689 11202 raft_consensus.cc:729] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f757e42d4c494588ad00add362c488ed, State: Initialized, Role: FOLLOWER
I20260924 20:08:53.932132 11201 ts_tablet_manager.cc:1428] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260924 20:08:53.932181 11203 consensus_queue.cc:260] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.932181 11202 consensus_queue.cc:260] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.932513 11184 heartbeater.cc:503] Master 127.0.115.190:34257 was elected leader, sending a full tablet report...
I20260924 20:08:53.932924 10916 heartbeater.cc:503] Master 127.0.115.190:34257 was elected leader, sending a full tablet report...
I20260924 20:08:53.933193 11202 ts_tablet_manager.cc:1428] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:08:53.933938 11050 heartbeater.cc:503] Master 127.0.115.190:34257 was elected leader, sending a full tablet report...
I20260924 20:08:53.935150 11203 ts_tablet_manager.cc:1428] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260924 20:08:53.952447 11208 raft_consensus.cc:482] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:08:53.952654 11208 raft_consensus.cc:504] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.953785 11208 leader_election.cc:288] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1659466ed922489d80c8d081346d6b97 (127.0.115.130:39879), 1ba08890fee24894b7b5ac64150c55c1 (127.0.115.131:44373)
I20260924 20:08:53.957615 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.954159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:48953 (local address 127.0.115.130:39879)
0924 20:08:53.954356 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.954361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.954383 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:08:53.954541 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.954544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.954626 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.954702 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:53.955044 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.955550 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.956659 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.957061 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.957200 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.957260 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.957425 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.957495 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 20:08:53.957628 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.954044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:39879 (local address 127.0.115.129:48953)
0924 20:08:53.954302 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.954315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.954436 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.954802 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.954805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.954824 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:53.954933 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.954938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.955659 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.955664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.956533 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.956543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.957184 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.957188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.957291 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.957322 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.957410 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.957481 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260924 20:08:53.957935 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.954558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:56495 (local address 127.0.115.131:44373)
0924 20:08:53.954724 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.954729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.954891 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:08:53.955029 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.955032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.955081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.955179 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:08:53.955437 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.955920 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.956988 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.957400 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.957528 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.957576 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.957761 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.957831 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 20:08:53.957952 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.954412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44373 (local address 127.0.115.129:56495)
0924 20:08:53.954802 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:53.954813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:08:53.954931 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:08:53.955189 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:08:53.955191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:08:53.955207 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:08:53.955328 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.955333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.956052 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.956056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.956865 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:08:53.956874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.957522 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.957526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:08:53.957622 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.957652 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:08:53.957745 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:08:53.957811 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":106,"threads_started":1}
I20260924 20:08:53.958280 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8dff97d3dd4eaab4150bfb1b13ba7c" candidate_uuid: "f757e42d4c494588ad00add362c488ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1659466ed922489d80c8d081346d6b97" is_pre_election: true
I20260924 20:08:53.958635 11004 raft_consensus.cc:2454] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f757e42d4c494588ad00add362c488ed in term 0.
I20260924 20:08:53.958622 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8dff97d3dd4eaab4150bfb1b13ba7c" candidate_uuid: "f757e42d4c494588ad00add362c488ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba08890fee24894b7b5ac64150c55c1" is_pre_election: true
I20260924 20:08:53.958906 11138 raft_consensus.cc:2454] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f757e42d4c494588ad00add362c488ed in term 0.
I20260924 20:08:53.959148 10803 leader_election.cc:302] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1659466ed922489d80c8d081346d6b97, f757e42d4c494588ad00add362c488ed; no voters:
I20260924 20:08:53.959438 11208 raft_consensus.cc:2781] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:08:53.959514 11208 raft_consensus.cc:482] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:08:53.959543 11208 raft_consensus.cc:3037] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:53.960675 11208 raft_consensus.cc:504] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.961023 11208 leader_election.cc:288] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [CANDIDATE]: Term 1 election: Requested vote from peers 1659466ed922489d80c8d081346d6b97 (127.0.115.130:39879), 1ba08890fee24894b7b5ac64150c55c1 (127.0.115.131:44373)
I20260924 20:08:53.961418 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8dff97d3dd4eaab4150bfb1b13ba7c" candidate_uuid: "f757e42d4c494588ad00add362c488ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1659466ed922489d80c8d081346d6b97"
I20260924 20:08:53.961515 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8dff97d3dd4eaab4150bfb1b13ba7c" candidate_uuid: "f757e42d4c494588ad00add362c488ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba08890fee24894b7b5ac64150c55c1"
I20260924 20:08:53.961567 11004 raft_consensus.cc:3037] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:53.961637 11138 raft_consensus.cc:3037] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:08:53.962618 11004 raft_consensus.cc:2454] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f757e42d4c494588ad00add362c488ed in term 1.
I20260924 20:08:53.962622 11138 raft_consensus.cc:2454] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f757e42d4c494588ad00add362c488ed in term 1.
I20260924 20:08:53.962932 10803 leader_election.cc:302] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba08890fee24894b7b5ac64150c55c1, f757e42d4c494588ad00add362c488ed; no voters:
I20260924 20:08:53.963155 11208 raft_consensus.cc:2781] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:08:53.963399 11208 raft_consensus.cc:686] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 1 LEADER]: Becoming Leader. State: Replica: f757e42d4c494588ad00add362c488ed, State: Running, Role: LEADER
I20260924 20:08:53.963775 11208 consensus_queue.cc:237] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } }
I20260924 20:08:53.967096 10725 catalog_manager.cc:5495] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed reported cstate change: term changed from 0 to 1, leader changed from <none> to f757e42d4c494588ad00add362c488ed (127.0.115.129). New cstate: current_term: 1 leader_uuid: "f757e42d4c494588ad00add362c488ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f757e42d4c494588ad00add362c488ed" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38425 } health_report { overall_health: HEALTHY } } }
I20260924 20:08:53.988277 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:53.984407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.0.115.129:38425)
0924 20:08:53.984570 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:08:53.984574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:08:53.984590 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:08:53.984839 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:08:53.984842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:08:53.984902 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:08:53.985031 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:08:53.985293 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.985806 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.986687 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:08:53.986917 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:08:53.987027 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:08:53.987360 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:08:53.987787 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:08:53.987928 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:08:53.988066 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 20:08:53.988125 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:08:53.994477 11004 raft_consensus.cc:1261] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 1 FOLLOWER]: Refusing update from remote peer f757e42d4c494588ad00add362c488ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:53.994477 11138 raft_consensus.cc:1261] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 1 FOLLOWER]: Refusing update from remote peer f757e42d4c494588ad00add362c488ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:08:53.995289 11212 consensus_queue.cc:1035] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1659466ed922489d80c8d081346d6b97" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:53.995462 11208 consensus_queue.cc:1035] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba08890fee24894b7b5ac64150c55c1" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:08:54.004653 11217 mvcc.cc:205] Tried to move back new op lower bound from 7332989067233542144 to 7332989067118727168. Current Snapshot: MvccSnapshot[applied={T|T < 7332989067233542144}]
I20260924 20:08:54.005663 11216 mvcc.cc:205] Tried to move back new op lower bound from 7332989067233542144 to 7332989067118727168. Current Snapshot: MvccSnapshot[applied={T|T < 7332989067233542144}]
I20260924 20:08:54.012670 10725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:54.012861 10725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:08:54.015367 10725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4a533515c8140bbaf681207fcd8884d: Sending DeleteTablet for 3 replicas of tablet cc8dff97d3dd4eaab4150bfb1b13ba7c
I20260924 20:08:54.016193 10984 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:54 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:54184
I20260924 20:08:54.016212 10850 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:54 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:56084
I20260924 20:08:54.016271 11118 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:08:54 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:51778
I20260924 20:08:54.016690 11225 tablet_replica.cc:330] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: stopping tablet replica
I20260924 20:08:54.016749 11226 tablet_replica.cc:330] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed: stopping tablet replica
I20260924 20:08:54.017030 11226 raft_consensus.cc:2229] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:08:54.017033 11225 raft_consensus.cc:2229] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:54.017349 11225 raft_consensus.cc:2258] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:54.017403 11226 raft_consensus.cc:2258] T cc8dff97d3dd4eaab4150bfb1b13ba7c P f757e42d4c494588ad00add362c488ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:54.017719 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10787
I20260924 20:08:54.017766 11227 tablet_replica.cc:330] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: stopping tablet replica
I20260924 20:08:54.017982 11227 raft_consensus.cc:2229] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:08:54.018200 11227 raft_consensus.cc:2258] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:08:54.020259 11227 ts_tablet_manager.cc:1905] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:54.020912 11225 ts_tablet_manager.cc:1905] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:08:54.023018 11227 ts_tablet_manager.cc:1918] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:54.023115 11227 log.cc:1196] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/cc8dff97d3dd4eaab4150bfb1b13ba7c
I20260924 20:08:54.023451 11227 ts_tablet_manager.cc:1939] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1659466ed922489d80c8d081346d6b97: Deleting consensus metadata
I20260924 20:08:54.023671 11225 ts_tablet_manager.cc:1918] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:08:54.023765 11225 log.cc:1196] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/cc8dff97d3dd4eaab4150bfb1b13ba7c
I20260924 20:08:54.024101 11225 ts_tablet_manager.cc:1939] T cc8dff97d3dd4eaab4150bfb1b13ba7c P 1ba08890fee24894b7b5ac64150c55c1: Deleting consensus metadata
I20260924 20:08:54.025094 10709 catalog_manager.cc:4841] TS 1659466ed922489d80c8d081346d6b97 (127.0.115.130:39879): tablet cc8dff97d3dd4eaab4150bfb1b13ba7c (table test-table [id=62d2bbfd14d04ed1a32ad06e4e3cb1f2]) successfully deleted
W20260924 20:08:54.025259 10712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:38425 (error 108)
I20260924 20:08:54.025332 10709 catalog_manager.cc:4841] TS 1ba08890fee24894b7b5ac64150c55c1 (127.0.115.131:44373): tablet cc8dff97d3dd4eaab4150bfb1b13ba7c (table test-table [id=62d2bbfd14d04ed1a32ad06e4e3cb1f2]) successfully deleted
I20260924 20:08:54.025453 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10921
I20260924 20:08:54.026930 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:08:54.026664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:38425 (local address 127.0.0.1:56102)
0924 20:08:54.026792 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 20:08:54.026860 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:38425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:08:54.027128 10712 catalog_manager.cc:4584] TS f757e42d4c494588ad00add362c488ed (127.0.115.129:38425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc8dff97d3dd4eaab4150bfb1b13ba7c: Network error: Client connection negotiation failed: client connection to 127.0.115.129:38425: connect: Connection refused (error 111)
I20260924 20:08:54.031435 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 11055
I20260924 20:08:54.037002 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 10693
2026-09-24T20:08:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2292 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:08:54 dist-test-slave-2pm7 krb5kdc[11231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:08:54 dist-test-slave-2pm7 krb5kdc[11231](info): set up 2 sockets
Sep 24 20:08:54 dist-test-slave-2pm7 krb5kdc[11231](info): commencing operation
krb5kdc: starting...
W20260924 20:08:55.069273 462 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:08:55 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:08:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/rangeradmin_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/rangeradmin_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/rangerlookup_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/rangerlookup_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
I20260924 20:08:55.230719 462 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIXC3Dg/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/postgres -l logfile start
2026-09-24 20:08:57.815 UTC [11261] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 20:08:57.815 UTC [11261] LOG: listening on IPv4 address "127.0.115.148", port 34769
2026-09-24 20:08:57.819 UTC [11261] LOG: listening on Unix socket "/tmp/.s.PGSQL.34769"
2026-09-24 20:08:57.857 UTC [11264] LOG: database system was shut down at 2026-09-24 20:08:56 UTC
2026-09-24 20:08:57.861 UTC [11261] LOG: database system is ready to accept connections
I20260924 20:08:58.845039 462 mini_postgres.cc:96] Postgres bound to 34769
2026-09-24 20:08:58.851 UTC [11272] FATAL: database "slave" does not exist
127.0.115.148:34769 - accepting connections
I20260924 20:08:58.852273 462 mini_ranger.cc:161] Starting Ranger...
I20260924 20:08:58.868083 462 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 20:08:59.399559 462 mini_ranger.cc:87] Created ranger Postgres database
I20260924 20:08:59.399663 462 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-admin
2026-09-24 20:08:59,798 [I] DB FLAVOR :POSTGRES
2026-09-24 20:08:59,799 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:08:59,799 [I] Checking connection..
2026-09-24 20:08:59,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 20:09:00,072 [I] Checking connection passed.
2026-09-24 20:09:00,072 [I] --------- Verifying version history table ---------
2026-09-24 20:09:00,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:09:00,304 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:09:00,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:09:00,550 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:09:00,551 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 20:09:00,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 20:09:00.810 UTC [11336] WARNING: there is no transaction in progress
2026-09-24 20:09:00,817 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 20:09:00,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:09:01,072 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 20:09:01,073 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:09:01,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 20:09:01,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 20:09:01,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2pm7.c.gcp-upstream.internal','N') ;"
2026-09-24 20:09:01,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:09:02,037 [I] Table x_portal_user does not exist in database ranger
2026-09-24 20:09:02,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:09:02,275 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 20:09:02,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:09:02,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:09:02,780 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 20:09:02,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 20:09:03.767 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.784 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.801 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.816 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.832 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.853 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.908 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.916 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.932 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.942 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.953 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.969 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.979 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.986 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:03.993 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:04.000 UTC [11444] WARNING: there is no transaction in progress
2026-09-24 20:09:04,691 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 20:09:04,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 20:09:04,942 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 20:09:04,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:09:05,187 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 20:09:05,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:09:05,422 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 20:09:05,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:09:05,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:09:05,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2pm7.c.gcp-upstream.internal';"
2026-09-24 20:09:06,139 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 20:09:06,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 20:09:06,377 [I] DB_PATCHES have already been applied
I20260924 20:09:06.383388 462 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIXC3Dg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 20:09:06.383464 462 mini_ranger.cc:193] Using host: 127.0.115.148
I20260924 20:09:06.387195 462 mini_ranger.cc:227] Ranger admin URL: http://127.0.115.148:42125
Sep 24, 2026 8:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 8:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 8:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 8:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 8:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 20:09:07.506948 462 mini_ranger.cc:160] Time spent starting Ranger: real 8.655s user 0.000s sys 0.012s
Sep 24, 2026 8:09:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 8:09:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 8:09:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 8:09:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 8:09:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 20:09:28.300187 462 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/rangerkms_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/rangerkms_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.115.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.115.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 20:09:28 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280568, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 20:09:28.392165 462 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 20:09:28.402037 462 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 20:09:29.406805 462 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 20:09:29.406908 462 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms
I20260924 20:09:29.407013 462 mini_ranger_kms.cc:227] Using postgres at 127.0.115.148:34769
2026-09-24 20:09:29,871 [I] DB FLAVOR :POSTGRES
2026-09-24 20:09:29,871 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:09:29,871 [I] Checking connection
2026-09-24 20:09:29,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 20:09:30,105 [I] connection success
2026-09-24 20:09:30,105 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 20:09:30,105 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 20:09:30,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 20:09:30,342 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 20:09:30,342 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:09:30,342 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 20:09:30,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:34769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 20:09:30,624 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 20:09:31.065651 462 mini_ranger_kms.cc:310] Created kms service
I20260924 20:09:31.258610 462 mini_ranger_kms.cc:326] Created kudu user
I20260924 20:09:31.303462 462 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 20:09:31 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280568, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:09:31.715770 462 mini_ranger_kms.cc:384] Added ranger policy
I20260924 20:09:31.715931 462 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 20:09:31.715991 462 mini_ranger_kms.cc:240] Using host: 127.0.115.148
I20260924 20:09:31.719662 462 mini_ranger_kms.cc:276] Ranger KMS PID: 11624
I20260924 20:09:31.719805 462 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.115.148:36491
Sep 24, 2026 8:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 8:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 8:09:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 8:09:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 20:09:33.984215 462 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.592s user 0.000s sys 0.199s
I20260924 20:09:33.984346 462 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 20:09:33.984457 462 mini_ranger_kms.cc:401] 127.0.115.148:36491/kms/v1/keys
Sep 24 20:09:34 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790280574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.115.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 20:09:34 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.115.148@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
Sep 24 20:09:34 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.115.148@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
Sep 24 20:09:37 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:09:37.220222 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46623
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:46623
--ranger_config_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:36491/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:09:37.333716 11675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:09:37.333999 11675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:09:37.334039 11675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:09:37.338331 11675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:09:37.338406 11675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:09:37.338425 11675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:09:37.338442 11675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:09:37.338456 11675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:09:37.344264 11675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.115.148:36491/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:46623
--ranger_config_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:09:37.345455 11675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:09:37.346863 11675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:09:37.357427 11675 server_base.cc:1034] running on GCE node
W20260924 20:09:37.357384 11680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:09:37.357378 11681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:09:37.357561 11683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:09:37.358343 11675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:09:37.359577 11675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:09:37.360761 11675 hybrid_clock.cc:648] HybridClock initialized: now 1790280577360732 us; error 47 us; skew 500 ppm
Sep 24 20:09:37 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:09:37.369774 11675 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:09:37.370937 11675 webserver.cc:460] Webserver started at http://127.0.115.190:42951/ using document root <none> and password file <none>
I20260924 20:09:37.371549 11675 fs_manager.cc:359] Metadata directory not provided
I20260924 20:09:37.371609 11675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:09:37.371836 11675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:09:37 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:09:37.426584 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "4dd42bb5c41d4ede97a92e855de3002c"
format_stamp: "Formatted at 2026-09-24 20:09:37 on dist-test-slave-2pm7"
server_key: "e05474262e1a89b0fa7195d181014c7f"
server_key_iv: "3da61eaee9ec66b1272e65d73109739f"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:37.427405 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "4dd42bb5c41d4ede97a92e855de3002c"
format_stamp: "Formatted at 2026-09-24 20:09:37 on dist-test-slave-2pm7"
server_key: "e05474262e1a89b0fa7195d181014c7f"
server_key_iv: "3da61eaee9ec66b1272e65d73109739f"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:37.432634 11675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 20:09:37.448172 11690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:37.449709 11675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:09:37.449892 11675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "4dd42bb5c41d4ede97a92e855de3002c"
format_stamp: "Formatted at 2026-09-24 20:09:37 on dist-test-slave-2pm7"
server_key: "e05474262e1a89b0fa7195d181014c7f"
server_key_iv: "3da61eaee9ec66b1272e65d73109739f"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:37.450037 11675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:09:37.469624 11675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:09:37.472864 11675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:09:37.473084 11675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:09:37.481362 11675 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:46623
I20260924 20:09:37.481428 11742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:46623 every 8 connection(s)
I20260924 20:09:37.482956 11675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:09:37.488361 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 11675
I20260924 20:09:37.488607 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:09:37.489161 462 external_mini_cluster.cc:1428] Setting key ca7e5e0c0430a39ad05bbffbab2b6655
I20260924 20:09:37.489571 11743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:09:37.497655 11743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c: Bootstrap starting.
Sep 24 20:09:37 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:09:37.500483 11743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c: Neither blocks nor log segments found. Creating new log.
I20260924 20:09:37.501806 11743 log.cc:824] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:09:37.504397 11743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c: No bootstrap required, opened a new log
I20260924 20:09:37.504877 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:37.492125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.0.115.190:46623)
0924 20:09:37.492621 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:37.492641 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:37.492689 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:09:37.493398 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:37.493414 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:37.493764 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:37.494126 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:09:37.494895 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:37.495899 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:37.496808 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:37.497298 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:37.497470 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:37.500059 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:09:37.500065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:09:37.500079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:09:37.500107 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:09:37.501763 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:37.502256 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:37.502265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:37.502275 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:37.502424 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:37.502831 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:37.502833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:37.502834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:37.503227 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:09:37.503445 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:37.503724 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 20:09:37.504070 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":164,"threads_started":1}
I20260924 20:09:37.508594 11743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd42bb5c41d4ede97a92e855de3002c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46623 } }
I20260924 20:09:37.508881 11743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:09:37.508956 11743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd42bb5c41d4ede97a92e855de3002c, State: Initialized, Role: FOLLOWER
I20260924 20:09:37.509449 11743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd42bb5c41d4ede97a92e855de3002c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46623 } }
I20260924 20:09:37.509608 11743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:09:37.509701 11743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:09:37.509815 11743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:09:37.510955 11743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd42bb5c41d4ede97a92e855de3002c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46623 } }
I20260924 20:09:37.511335 11743 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd42bb5c41d4ede97a92e855de3002c; no voters:
I20260924 20:09:37.511682 11743 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:09:37.511806 11748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:09:37.512096 11748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [term 1 LEADER]: Becoming Leader. State: Replica: 4dd42bb5c41d4ede97a92e855de3002c, State: Running, Role: LEADER
I20260924 20:09:37.512508 11748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd42bb5c41d4ede97a92e855de3002c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46623 } }
I20260924 20:09:37.513787 11748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd42bb5c41d4ede97a92e855de3002c. Latest consensus state: current_term: 1 leader_uuid: "4dd42bb5c41d4ede97a92e855de3002c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd42bb5c41d4ede97a92e855de3002c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46623 } } }
I20260924 20:09:37.514016 11748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [sys.catalog]: This master's current role is: LEADER
I20260924 20:09:37.514242 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dd42bb5c41d4ede97a92e855de3002c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd42bb5c41d4ede97a92e855de3002c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46623 } } }
I20260924 20:09:37.514339 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [sys.catalog]: This master's current role is: LEADER
I20260924 20:09:37.515094 11743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:09:37.515334 11752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:09:37.518416 11752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:09:37.518908 11743 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:09:37.530751 11752 catalog_manager.cc:1347] Generated new cluster ID: ca82df47052948b681f157ce25f1bce8
I20260924 20:09:37.530838 11752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:09:37.547323 11752 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:09:37.548754 11752 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:09:37.569358 11752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c: Generated new TSK 0
I20260924 20:09:37.571357 11752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:09:38.331578 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:36491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46623
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:09:38.478102 11788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:09:38.478417 11788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:09:38.478513 11788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:09:38.498066 11788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:09:38.498245 11788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:09:38.498401 11788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:09:38.506858 11788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.115.148:36491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:09:38.508481 11788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:09:38.509984 11788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:09:38.523752 11795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:09:38.523913 11788 server_base.cc:1034] running on GCE node
W20260924 20:09:38.523784 11798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:09:38.523752 11796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:09:38.524680 11788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:09:38.525353 11788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:09:38.526544 11788 hybrid_clock.cc:648] HybridClock initialized: now 1790280578526512 us; error 54 us; skew 500 ppm
Sep 24 20:09:38 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:09:38.530124 11788 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:09:38.531154 11788 webserver.cc:460] Webserver started at http://127.0.115.129:43667/ using document root <none> and password file <none>
I20260924 20:09:38.531744 11788 fs_manager.cc:359] Metadata directory not provided
I20260924 20:09:38.531836 11788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:09:38.532167 11788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:09:38 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:09:38.546820 11788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "92f5e38932cf4024b095d9bdc680ce74"
format_stamp: "Formatted at 2026-09-24 20:09:38 on dist-test-slave-2pm7"
server_key: "29d831c7f6e77a5887f7450aa733a034"
server_key_iv: "990ae1a5858420cb795f87160c333750"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:38.547678 11788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "92f5e38932cf4024b095d9bdc680ce74"
format_stamp: "Formatted at 2026-09-24 20:09:38 on dist-test-slave-2pm7"
server_key: "29d831c7f6e77a5887f7450aa733a034"
server_key_iv: "990ae1a5858420cb795f87160c333750"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:38.553934 11788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260924 20:09:38.584417 11805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:38.586421 11788 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260924 20:09:38.586618 11788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "92f5e38932cf4024b095d9bdc680ce74"
format_stamp: "Formatted at 2026-09-24 20:09:38 on dist-test-slave-2pm7"
server_key: "29d831c7f6e77a5887f7450aa733a034"
server_key_iv: "990ae1a5858420cb795f87160c333750"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:38.586853 11788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:09:38.622063 11788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:09:38.626364 11788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:09:38.626645 11788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:09:38.627465 11788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:09:38.628706 11788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:09:38.628836 11788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:38.628994 11788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:09:38.629092 11788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:38.644150 11788 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:39955
I20260924 20:09:38.645596 11788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:09:38.646873 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 11788
I20260924 20:09:38.647017 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:09:38.647281 11920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:39955 every 8 connection(s)
I20260924 20:09:38.647419 462 external_mini_cluster.cc:1428] Setting key 03f21beddccd5072addd6f208d198a1e
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Sep 24 20:09:38 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:09:38.671774 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:38.650111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46623 (local address 127.0.115.129:55363)
0924 20:09:38.651433 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:38.651515 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:38.652818 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:38.653894 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:38.653907 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:38.654321 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:09:38.655131 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:38.655156 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.656634 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.656640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:38.658338 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:38.658353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.659217 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.659224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:38.659455 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:38.660216 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:09:38.660245 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:09:38.665016 (+ 4771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:09:38.667513 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:09:38.667522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:09:38.667544 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:09:38.667985 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:09:38.668346 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:09:38.668352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:09:38.668357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:09:38.668560 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:09:38.669536 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:09:38.669545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:09:38.669919 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 20:09:38.670165 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 20:09:38.670561 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"thread_start_us":127,"threads_started":1}
I20260924 20:09:38.673271 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:38.649486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:55363 (local address 127.0.115.190:46623)
0924 20:09:38.653273 (+ 3787us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:38.653280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:38.653297 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:09:38.653346 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:38.653353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:38.653417 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:38.653537 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:09:38.655330 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.656418 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.658495 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.658884 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.659007 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:38.665262 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:09:38.665267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:09:38.665272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:09:38.665306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:09:38.667349 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:38.668124 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:38.668128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:38.668130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:38.668231 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:38.668688 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:38.668693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:38.668696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:38.668980 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:09:38.669465 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:38.672265 (+ 2800us) server_negotiation.cc:299] Negotiation successful
0924 20:09:38.672404 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3682,"thread_start_us":96,"threads_started":1}
I20260924 20:09:38.674328 11921 heartbeater.cc:347] Connected to a master server at 127.0.115.190:46623
I20260924 20:09:38.674603 11921 heartbeater.cc:464] Registering TS with master...
I20260924 20:09:38.675161 11921 heartbeater.cc:511] Master 127.0.115.190:46623 requested a full tablet report, sending...
I20260924 20:09:38.676818 11707 ts_manager.cc:194] Registered new tserver with Master: 92f5e38932cf4024b095d9bdc680ce74 (127.0.115.129:39955)
I20260924 20:09:38.678159 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:55363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:09:38.684296 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:36491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46623
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:09:38 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 20:09:38.812107 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:09:38.812395 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:09:38.812492 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:09:38.816802 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:09:38.816883 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:09:38.816982 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:09:38.822778 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.115.148:36491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:09:38.823907 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:09:38.824997 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:09:38.834890 11933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:09:38.834904 11936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:09:38.835167 11927 server_base.cc:1034] running on GCE node
W20260924 20:09:38.834870 11934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:09:38.835803 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:09:38.836591 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:09:38.837848 11927 hybrid_clock.cc:648] HybridClock initialized: now 1790280578837813 us; error 47 us; skew 500 ppm
Sep 24 20:09:38 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:09:38.841357 11927 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:09:38.842520 11927 webserver.cc:460] Webserver started at http://127.0.115.130:40917/ using document root <none> and password file <none>
I20260924 20:09:38.843310 11927 fs_manager.cc:359] Metadata directory not provided
I20260924 20:09:38.843415 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:09:38.843649 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:09:38 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:09:38.857555 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "52172a52271f42108ab9f549ecc7e995"
format_stamp: "Formatted at 2026-09-24 20:09:38 on dist-test-slave-2pm7"
server_key: "f02660743c6721a011c65b66469d2e8e"
server_key_iv: "9938e4336b46c19daffbd83800d55b9b"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:38.858197 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "52172a52271f42108ab9f549ecc7e995"
format_stamp: "Formatted at 2026-09-24 20:09:38 on dist-test-slave-2pm7"
server_key: "f02660743c6721a011c65b66469d2e8e"
server_key_iv: "9938e4336b46c19daffbd83800d55b9b"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:38.862496 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:09:38.888253 11943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:38.890102 11927 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260924 20:09:38.890250 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "52172a52271f42108ab9f549ecc7e995"
format_stamp: "Formatted at 2026-09-24 20:09:38 on dist-test-slave-2pm7"
server_key: "f02660743c6721a011c65b66469d2e8e"
server_key_iv: "9938e4336b46c19daffbd83800d55b9b"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:38.890395 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:09:38.905527 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:09:38.909526 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:09:38.909741 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:09:38.910456 11927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:09:38.911513 11927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:09:38.911578 11927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:38.911646 11927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:09:38.911679 11927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:38.923002 11927 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:36987
I20260924 20:09:38.924291 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:09:38.926174 12056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:36987 every 8 connection(s)
I20260924 20:09:38.930364 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 11927
I20260924 20:09:38.930521 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:09:38.930878 462 external_mini_cluster.cc:1428] Setting key da0c4a5e164d0b8a3bec714c6cb704a4
Sep 24 20:09:38 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:09:38.953402 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:38.928205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:50415 (local address 127.0.115.190:46623)
0924 20:09:38.928338 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:38.928343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:38.932513 (+ 4170us) server_negotiation.cc:407] Connection header received
0924 20:09:38.936034 (+ 3521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:38.936038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:38.936097 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:38.936209 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:09:38.937921 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.938788 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.940651 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.941000 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.941091 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:38.948191 (+ 7100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:09:38.948196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:09:38.948199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:09:38.948232 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:09:38.950210 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:38.951353 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:38.951357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:38.951359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:38.951457 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:38.951912 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:38.951916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:38.951918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:38.952178 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:09:38.952263 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:38.953060 (+ 797us) server_negotiation.cc:299] Negotiation successful
0924 20:09:38.953223 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 20:09:38.954804 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:38.929062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46623 (local address 127.0.115.130:50415)
0924 20:09:38.932212 (+ 3150us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:38.932419 (+ 207us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:38.935700 (+ 3281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:38.936567 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:38.936580 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:38.936973 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:09:38.937736 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:38.937759 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.938962 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.938966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:38.940501 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:38.940514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:38.941340 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:38.941348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:38.941559 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:38.942534 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:09:38.942556 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:09:38.947889 (+ 5333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:09:38.950813 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:09:38.950822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:09:38.950843 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:09:38.951224 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:09:38.951569 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:09:38.951572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:09:38.951574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:09:38.951797 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:09:38.952336 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:09:38.952343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:09:38.952726 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 20:09:38.952952 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 20:09:38.953342 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2900,"thread_start_us":166,"threads_started":1}
I20260924 20:09:38.956141 12057 heartbeater.cc:347] Connected to a master server at 127.0.115.190:46623
I20260924 20:09:38.956400 12057 heartbeater.cc:464] Registering TS with master...
I20260924 20:09:38.956988 12057 heartbeater.cc:511] Master 127.0.115.190:46623 requested a full tablet report, sending...
I20260924 20:09:38.958263 11707 ts_manager.cc:194] Registered new tserver with Master: 52172a52271f42108ab9f549ecc7e995 (127.0.115.130:36987)
I20260924 20:09:38.959064 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:50415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:09:38.981434 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:36491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46623
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:09:39.153234 12062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:09:39.153692 12062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:09:39.153803 12062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:09:39.159956 12062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:09:39.160118 12062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:09:39.160249 12062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:09:39.168758 12062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.115.148:36491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:09:39.170538 12062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:09:39.172112 12062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:09:39.187211 12067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:09:39.187881 12062 server_base.cc:1034] running on GCE node
W20260924 20:09:39.188105 12070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:09:39.188421 12068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:09:39.188995 12062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:09:39.189862 12062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:09:39.194394 12062 hybrid_clock.cc:648] HybridClock initialized: now 1790280579194313 us; error 86 us; skew 500 ppm
Sep 24 20:09:39 dist-test-slave-2pm7 krb5kdc[11231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:09:39.200434 12062 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:09:39.201604 12062 webserver.cc:460] Webserver started at http://127.0.115.131:36647/ using document root <none> and password file <none>
I20260924 20:09:39.202355 12062 fs_manager.cc:359] Metadata directory not provided
I20260924 20:09:39.205353 12062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:09:39.205700 12062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:09:39 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:09:39.235666 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "5001727bcedc4d1e9fc4d12aaa820b16"
format_stamp: "Formatted at 2026-09-24 20:09:39 on dist-test-slave-2pm7"
server_key: "52fc1083182e522078b95df5108eaaa4"
server_key_iv: "4814dbb09b604377e3cdc2ff343e6092"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:39.236608 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "5001727bcedc4d1e9fc4d12aaa820b16"
format_stamp: "Formatted at 2026-09-24 20:09:39 on dist-test-slave-2pm7"
server_key: "52fc1083182e522078b95df5108eaaa4"
server_key_iv: "4814dbb09b604377e3cdc2ff343e6092"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:39.244129 12062 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
Sep 24 20:09:39 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:09:39.274093 12077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:39.280170 12062 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.003s
I20260924 20:09:39.280418 12062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "5001727bcedc4d1e9fc4d12aaa820b16"
format_stamp: "Formatted at 2026-09-24 20:09:39 on dist-test-slave-2pm7"
server_key: "52fc1083182e522078b95df5108eaaa4"
server_key_iv: "4814dbb09b604377e3cdc2ff343e6092"
server_key_version: "kuduclusterkey@0"
I20260924 20:09:39.280666 12062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:09:39.328863 12062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:09:39.332749 12062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:09:39.332993 12062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:09:39.333756 12062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:09:39.334815 12062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:09:39.334918 12062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:39.335016 12062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:09:39.335076 12062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:09:39.348817 12062 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:38797
I20260924 20:09:39.350134 12062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:09:39.351495 12190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:38797 every 8 connection(s)
I20260924 20:09:39.360250 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 12062
I20260924 20:09:39.360390 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:09:39.360744 462 external_mini_cluster.cc:1428] Setting key 78d63aa93204780a529377df3aa4808e
Sep 24 20:09:39 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:09:39.371873 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.355586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:54087 (local address 127.0.115.190:46623)
0924 20:09:39.355890 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.355895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.356538 (+ 643us) server_negotiation.cc:407] Connection header received
0924 20:09:39.357769 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.357773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.357832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.357884 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:09:39.359675 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.360510 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.364298 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.364630 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.364703 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.367712 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:09:39.367716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:09:39.367719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:09:39.367746 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:09:39.369478 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:39.370092 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:39.370096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:39.370098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:39.370191 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:39.370559 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:39.370562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:39.370564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:39.370789 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:09:39.370861 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.371569 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.371705 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260924 20:09:39.375697 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.354399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:46623 (local address 127.0.115.131:54087)
0924 20:09:39.356376 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.356432 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:39.357515 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:39.358434 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:39.358444 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:39.358785 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:09:39.359512 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.359532 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.360762 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.360766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.364160 (+ 3394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.364173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.364906 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.364909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.365120 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.365796 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:09:39.365818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:09:39.367517 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:09:39.369624 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:09:39.369630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:09:39.369649 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:09:39.369975 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:09:39.370295 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:09:39.370298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:09:39.370300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:09:39.370464 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:09:39.370943 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:09:39.370949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:09:39.371283 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 20:09:39.372423 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0924 20:09:39.372788 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1754,"thread_start_us":144,"threads_started":1}
I20260924 20:09:39.377290 12191 heartbeater.cc:347] Connected to a master server at 127.0.115.190:46623
I20260924 20:09:39.377547 12191 heartbeater.cc:464] Registering TS with master...
I20260924 20:09:39.378216 12191 heartbeater.cc:511] Master 127.0.115.190:46623 requested a full tablet report, sending...
I20260924 20:09:39.379456 11707 ts_manager.cc:194] Registered new tserver with Master: 5001727bcedc4d1e9fc4d12aaa820b16 (127.0.115.131:38797)
I20260924 20:09:39.380230 11707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:54087
I20260924 20:09:39.386309 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:09:39.403935 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.392551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.0.115.190:46623)
0924 20:09:39.392672 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.392677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.393091 (+ 414us) server_negotiation.cc:407] Connection header received
0924 20:09:39.393359 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.393363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.393415 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.393471 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:09:39.394620 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.395564 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.397186 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.397585 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.397694 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.399812 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:09:39.399817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:09:39.399820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:09:39.399854 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:09:39.401800 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:39.402365 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:39.402369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:39.402371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:39.402465 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:09:39.402822 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:09:39.402826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:09:39.402828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:09:39.403075 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:09:39.403189 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.403608 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.403746 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:09:39.408134 11707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:09:39.413080 11707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:09:39.435415 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.424582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39955 (local address 127.0.0.1:36686)
0924 20:09:39.426188 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.426246 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:39.426384 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:39.427542 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:39.427549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:39.427579 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:09:39.427758 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.427767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.431872 (+ 4105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.431876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.433314 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.433326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.434781 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.434785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.434921 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.434964 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 20:09:39.435097 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 20:09:39.435239 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"thread_start_us":116,"threads_started":1}
I20260924 20:09:39.435917 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.425929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.0.115.129:39955)
0924 20:09:39.427081 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.427088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.427108 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:09:39.427173 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.427177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.427347 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.427519 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:09:39.427906 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.431496 (+ 3590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.433468 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.434432 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.434548 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.434712 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.435643 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.435755 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":99,"threads_started":1}
I20260924 20:09:39.437541 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.425443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:36987 (local address 127.0.0.1:38788)
0924 20:09:39.427998 (+ 2555us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.428022 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:39.428121 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:39.428817 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:39.428820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:39.428838 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:09:39.428935 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.428942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.430444 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.430448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.433955 (+ 3507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.433966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.437142 (+ 3176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.437146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.437250 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.437273 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:09:39.437343 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 20:09:39.437408 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2467,"mutex_wait_us":2176,"thread_start_us":108,"threads_started":1}
I20260924 20:09:39.437992 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.425599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.0.115.130:36987)
0924 20:09:39.427797 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.427812 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.428221 (+ 409us) server_negotiation.cc:407] Connection header received
0924 20:09:39.428305 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.428312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.428514 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.428664 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:09:39.429093 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.430301 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.435178 (+ 4877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.436822 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.436928 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.437065 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.437720 (+ 655us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.437850 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2105,"thread_start_us":112,"threads_started":1}
I20260924 20:09:39.439512 11854 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac4c81b91624f58b23745f8125a6621 (DEFAULT_TABLE table=test-table [id=d033da3c52cd4ba7ba1e716e98282ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:09:39.440847 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac4c81b91624f58b23745f8125a6621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:09:39.440872 11991 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac4c81b91624f58b23745f8125a6621 (DEFAULT_TABLE table=test-table [id=d033da3c52cd4ba7ba1e716e98282ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:09:39.441941 11991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac4c81b91624f58b23745f8125a6621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:09:39.450278 12209 tablet_bootstrap.cc:492] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74: Bootstrap starting.
I20260924 20:09:39.451045 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.424976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:38797 (local address 127.0.0.1:48634)
0924 20:09:39.426876 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.426890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:39.427000 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:39.427901 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:39.427903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:39.427920 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:09:39.428014 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.428019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.430624 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.430627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.432017 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.432029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.450593 (+ 18564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.450597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.450717 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.450749 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 20:09:39.450826 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:09:39.450907 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"thread_start_us":109,"threads_started":1}
I20260924 20:09:39.451531 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.425501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.0.115.131:38797)
0924 20:09:39.426719 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.426733 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.427016 (+ 283us) server_negotiation.cc:407] Connection header received
0924 20:09:39.427113 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.427119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.427348 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.427525 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:09:39.428136 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.430481 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.432197 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.434845 (+ 2648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.434948 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.435081 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.451264 (+ 16183us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.451393 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":108,"threads_started":1}
I20260924 20:09:39.453001 12209 tablet_bootstrap.cc:654] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74: Neither blocks nor log segments found. Creating new log.
I20260924 20:09:39.454079 12209 log.cc:824] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74: Log is configured to *not* fsync() on all Append() calls
I20260924 20:09:39.455332 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac4c81b91624f58b23745f8125a6621 (DEFAULT_TABLE table=test-table [id=d033da3c52cd4ba7ba1e716e98282ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:09:39.456473 12209 tablet_bootstrap.cc:492] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74: No bootstrap required, opened a new log
I20260924 20:09:39.456687 12209 ts_tablet_manager.cc:1397] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:09:39.456436 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac4c81b91624f58b23745f8125a6621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:09:39.462096 12209 raft_consensus.cc:348] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.462435 12209 raft_consensus.cc:374] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:09:39.462527 12209 raft_consensus.cc:729] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f5e38932cf4024b095d9bdc680ce74, State: Initialized, Role: FOLLOWER
I20260924 20:09:39.463096 12209 consensus_queue.cc:260] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.464322 12209 ts_tablet_manager.cc:1428] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 20:09:39.464792 11921 heartbeater.cc:503] Master 127.0.115.190:46623 was elected leader, sending a full tablet report...
I20260924 20:09:39.465338 12210 tablet_bootstrap.cc:492] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995: Bootstrap starting.
I20260924 20:09:39.466760 12213 tablet_bootstrap.cc:492] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Bootstrap starting.
I20260924 20:09:39.468066 12210 tablet_bootstrap.cc:654] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995: Neither blocks nor log segments found. Creating new log.
I20260924 20:09:39.469519 12213 tablet_bootstrap.cc:654] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Neither blocks nor log segments found. Creating new log.
I20260924 20:09:39.469512 12210 log.cc:824] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995: Log is configured to *not* fsync() on all Append() calls
I20260924 20:09:39.470381 12213 log.cc:824] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Log is configured to *not* fsync() on all Append() calls
I20260924 20:09:39.471879 12210 tablet_bootstrap.cc:492] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995: No bootstrap required, opened a new log
I20260924 20:09:39.472142 12210 ts_tablet_manager.cc:1397] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260924 20:09:39.472307 12213 tablet_bootstrap.cc:492] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: No bootstrap required, opened a new log
I20260924 20:09:39.472468 12213 ts_tablet_manager.cc:1397] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 20:09:39.476714 12213 raft_consensus.cc:348] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.476675 12210 raft_consensus.cc:348] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.476965 12213 raft_consensus.cc:374] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:09:39.476974 12210 raft_consensus.cc:374] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:09:39.477016 12213 raft_consensus.cc:729] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5001727bcedc4d1e9fc4d12aaa820b16, State: Initialized, Role: FOLLOWER
I20260924 20:09:39.477052 12210 raft_consensus.cc:729] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52172a52271f42108ab9f549ecc7e995, State: Initialized, Role: FOLLOWER
I20260924 20:09:39.477532 12213 consensus_queue.cc:260] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.477536 12210 consensus_queue.cc:260] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.478684 12210 ts_tablet_manager.cc:1428] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:09:39.478710 12213 ts_tablet_manager.cc:1428] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 20:09:39.479064 12057 heartbeater.cc:503] Master 127.0.115.190:46623 was elected leader, sending a full tablet report...
I20260924 20:09:39.479897 12191 heartbeater.cc:503] Master 127.0.115.190:46623 was elected leader, sending a full tablet report...
Sep 24 20:09:39 dist-test-slave-2pm7 krb5kdc[11231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
W20260924 20:09:39.606374 12192 tablet.cc:2369] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:09:39.629487 12217 raft_consensus.cc:482] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:09:39.629729 12217 raft_consensus.cc:504] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.631233 12217 leader_election.cc:288] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f5e38932cf4024b095d9bdc680ce74 (127.0.115.129:39955), 5001727bcedc4d1e9fc4d12aaa820b16 (127.0.115.131:38797)
I20260924 20:09:39.639684 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.632584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:43745 (local address 127.0.115.129:39955)
0924 20:09:39.632714 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.632719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.633403 (+ 684us) server_negotiation.cc:407] Connection header received
0924 20:09:39.633624 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.633628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.633677 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.633726 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:09:39.634632 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.635434 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.638205 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.638843 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.638948 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.639011 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.639418 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.639510 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:09:39.640154 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.632072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:48807 (local address 127.0.115.131:38797)
0924 20:09:39.632198 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.632203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.632865 (+ 662us) server_negotiation.cc:407] Connection header received
0924 20:09:39.633075 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.633081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.633133 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.633181 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:09:39.634283 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.635074 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.638662 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.639278 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.639373 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.639420 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.639911 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.640031 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:09:39.640594 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.632450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39955 (local address 127.0.115.130:43745)
0924 20:09:39.633321 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.633335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:39.633474 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:39.634071 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:39.634074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:39.634094 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:09:39.634492 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.634504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.635600 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.635604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.638036 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.638047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.639143 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.639147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.639265 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.639290 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:09:39.640349 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0924 20:09:39.640431 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260924 20:09:39.641521 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.631615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:38797 (local address 127.0.115.130:48807)
0924 20:09:39.632773 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.632790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:09:39.632950 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:09:39.633753 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:09:39.633756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:09:39.633777 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:09:39.633906 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.633913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.636010 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.636013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.638495 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:09:39.638507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.639661 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.639665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:09:39.639768 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.639788 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:09:39.640820 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0924 20:09:39.640896 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":31,"thread_start_us":124,"threads_started":1}
I20260924 20:09:39.641693 11859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac4c81b91624f58b23745f8125a6621" candidate_uuid: "52172a52271f42108ab9f549ecc7e995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f5e38932cf4024b095d9bdc680ce74" is_pre_election: true
I20260924 20:09:39.642059 11859 raft_consensus.cc:2454] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52172a52271f42108ab9f549ecc7e995 in term 0.
I20260924 20:09:39.643098 12131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac4c81b91624f58b23745f8125a6621" candidate_uuid: "52172a52271f42108ab9f549ecc7e995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" is_pre_election: true
I20260924 20:09:39.643419 12131 raft_consensus.cc:2454] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52172a52271f42108ab9f549ecc7e995 in term 0.
I20260924 20:09:39.644284 11947 leader_election.cc:302] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52172a52271f42108ab9f549ecc7e995, 92f5e38932cf4024b095d9bdc680ce74; no voters:
I20260924 20:09:39.644620 12217 raft_consensus.cc:2781] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:09:39.644694 12217 raft_consensus.cc:482] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:09:39.644733 12217 raft_consensus.cc:3037] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:09:39.646055 12217 raft_consensus.cc:504] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.646422 12217 leader_election.cc:288] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [CANDIDATE]: Term 1 election: Requested vote from peers 92f5e38932cf4024b095d9bdc680ce74 (127.0.115.129:39955), 5001727bcedc4d1e9fc4d12aaa820b16 (127.0.115.131:38797)
W20260924 20:09:39.647913 11922 tablet.cc:2369] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:09:39.647962 11859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac4c81b91624f58b23745f8125a6621" candidate_uuid: "52172a52271f42108ab9f549ecc7e995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f5e38932cf4024b095d9bdc680ce74"
I20260924 20:09:39.648092 11859 raft_consensus.cc:3037] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:09:39.649384 11859 raft_consensus.cc:2454] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52172a52271f42108ab9f549ecc7e995 in term 1.
I20260924 20:09:39.649760 11947 leader_election.cc:302] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52172a52271f42108ab9f549ecc7e995, 92f5e38932cf4024b095d9bdc680ce74; no voters:
I20260924 20:09:39.650070 12131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac4c81b91624f58b23745f8125a6621" candidate_uuid: "52172a52271f42108ab9f549ecc7e995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5001727bcedc4d1e9fc4d12aaa820b16"
I20260924 20:09:39.650200 12131 raft_consensus.cc:3037] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:09:39.651523 12131 raft_consensus.cc:2454] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52172a52271f42108ab9f549ecc7e995 in term 1.
I20260924 20:09:39.652324 12217 raft_consensus.cc:2781] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:09:39.652562 12217 raft_consensus.cc:686] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [term 1 LEADER]: Becoming Leader. State: Replica: 52172a52271f42108ab9f549ecc7e995, State: Running, Role: LEADER
I20260924 20:09:39.652954 12217 consensus_queue.cc:237] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } }
I20260924 20:09:39.657132 11707 catalog_manager.cc:5495] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52172a52271f42108ab9f549ecc7e995 (127.0.115.130). New cstate: current_term: 1 leader_uuid: "52172a52271f42108ab9f549ecc7e995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52172a52271f42108ab9f549ecc7e995" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 } health_report { overall_health: UNKNOWN } } }
I20260924 20:09:39.674124 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.669115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.0.115.130:36987)
0924 20:09:39.669245 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:09:39.669249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:09:39.669263 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:09:39.669343 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:09:39.669346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:09:39.669411 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:09:39.669516 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:09:39.669808 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.670633 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.671918 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:09:39.672290 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:09:39.672406 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:09:39.672765 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:09:39.673343 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:09:39.673461 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:09:39.673922 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 20:09:39.673984 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 20:09:39.681706 12131 raft_consensus.cc:1261] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 1 FOLLOWER]: Refusing update from remote peer 52172a52271f42108ab9f549ecc7e995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 20:09:39.682482 12058 tablet.cc:2369] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:09:39.682578 12217 consensus_queue.cc:1035] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5001727bcedc4d1e9fc4d12aaa820b16" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:09:39.683292 11859 raft_consensus.cc:1261] T 1ac4c81b91624f58b23745f8125a6621 P 92f5e38932cf4024b095d9bdc680ce74 [term 1 FOLLOWER]: Refusing update from remote peer 52172a52271f42108ab9f549ecc7e995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:09:39.683892 12217 consensus_queue.cc:1035] T 1ac4c81b91624f58b23745f8125a6621 P 52172a52271f42108ab9f549ecc7e995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f5e38932cf4024b095d9bdc680ce74" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:09:39.690508 12225 mvcc.cc:205] Tried to move back new op lower bound from 7332989254368145408 to 7332989254263742464. Current Snapshot: MvccSnapshot[applied={T|T < 7332989254368145408}]
I20260924 20:09:39.695417 12226 mvcc.cc:205] Tried to move back new op lower bound from 7332989254368145408 to 7332989254263742464. Current Snapshot: MvccSnapshot[applied={T|T < 7332989254368145408}]
I20260924 20:09:39.699227 12228 mvcc.cc:205] Tried to move back new op lower bound from 7332989254368145408 to 7332989254263742464. Current Snapshot: MvccSnapshot[applied={T|T < 7332989254368145408}]
I20260924 20:09:39.701000 11706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:09:39.701164 11706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:09:39.703850 11706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dd42bb5c41d4ede97a92e855de3002c: Sending DeleteTablet for 3 replicas of tablet 1ac4c81b91624f58b23745f8125a6621
I20260924 20:09:39.704895 12125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac4c81b91624f58b23745f8125a6621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:09:39 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:48634
I20260924 20:09:39.705789 12234 tablet_replica.cc:330] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: stopping tablet replica
I20260924 20:09:39.705848 11854 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac4c81b91624f58b23745f8125a6621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:09:39 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:36686
I20260924 20:09:39.706832 12234 raft_consensus.cc:2229] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:09:39.707000 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 11788
I20260924 20:09:39.707198 12234 raft_consensus.cc:2258] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:09:39.707844 11991 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac4c81b91624f58b23745f8125a6621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:09:39 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:38788
I20260924 20:09:39.708887 12234 ts_tablet_manager.cc:1905] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:09:39.712164 12234 ts_tablet_manager.cc:1918] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:09:39.712296 12234 log.cc:1196] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/1ac4c81b91624f58b23745f8125a6621
I20260924 20:09:39.712648 12234 ts_tablet_manager.cc:1939] T 1ac4c81b91624f58b23745f8125a6621 P 5001727bcedc4d1e9fc4d12aaa820b16: Deleting consensus metadata
I20260924 20:09:39.713965 11692 catalog_manager.cc:4841] TS 5001727bcedc4d1e9fc4d12aaa820b16 (127.0.115.131:38797): tablet 1ac4c81b91624f58b23745f8125a6621 (table test-table [id=d033da3c52cd4ba7ba1e716e98282ee7]) successfully deleted
W20260924 20:09:39.719630 11694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:39955 (error 108)
I20260924 20:09:39.721202 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.720880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39955 (local address 127.0.0.1:36688)
0924 20:09:39.721051 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.721115 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:39955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 20:09:39.721498 11694 catalog_manager.cc:4584] TS 92f5e38932cf4024b095d9bdc680ce74 (127.0.115.129:39955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ac4c81b91624f58b23745f8125a6621: Network error: Client connection negotiation failed: client connection to 127.0.115.129:39955: connect: Connection refused (error 111)
I20260924 20:09:39.721882 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 11927
I20260924 20:09:39.744877 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 12062
I20260924 20:09:39.746028 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:09:39.745652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:36987 (local address 127.0.0.1:38818)
0924 20:09:39.745834 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 20:09:39.745910 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.130:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 20:09:39.752843 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 11675
I20260924 20:09:39.759025 462 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 20:09:40.103829 462 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 20:09:40.104005 462 mini_ranger.cc:66] Stopping Ranger...
I20260924 20:09:40.448770 462 mini_ranger.cc:68] Stopped Ranger
2026-09-24 20:09:40.448 UTC [11261] LOG: received smart shutdown request
2026-09-24 20:09:40.452 UTC [11261] LOG: background worker "logical replication launcher" (PID 11270) exited with exit code 1
2026-09-24 20:09:40.453 UTC [11265] LOG: shutting down
2026-09-24 20:09:40.637 UTC [11261] LOG: database system is shut down
2026-09-24T20:09:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46715 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:09:40 dist-test-slave-2pm7 krb5kdc[12246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:09:40 dist-test-slave-2pm7 krb5kdc[12246](info): set up 2 sockets
Sep 24 20:09:40 dist-test-slave-2pm7 krb5kdc[12246](info): commencing operation
krb5kdc: starting...
W20260924 20:09:41.808288 462 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:09:41 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:09:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/rangeradmin_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/rangeradmin_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/rangerlookup_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/rangerlookup_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
I20260924 20:09:41.974421 462 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIXC3Dg/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/postgres -l logfile start
2026-09-24 20:09:44.515 UTC [12276] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 20:09:44.515 UTC [12276] LOG: listening on IPv4 address "127.0.115.148", port 44191
2026-09-24 20:09:44.519 UTC [12276] LOG: listening on Unix socket "/tmp/.s.PGSQL.44191"
2026-09-24 20:09:44.541 UTC [12279] LOG: database system was shut down at 2026-09-24 20:09:43 UTC
2026-09-24 20:09:44.544 UTC [12276] LOG: database system is ready to accept connections
I20260924 20:09:45.500093 462 mini_postgres.cc:96] Postgres bound to 44191
2026-09-24 20:09:45.506 UTC [12287] FATAL: database "slave" does not exist
127.0.115.148:44191 - accepting connections
I20260924 20:09:45.507025 462 mini_ranger.cc:161] Starting Ranger...
I20260924 20:09:45.522253 462 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 20:09:46.017127 462 mini_ranger.cc:87] Created ranger Postgres database
I20260924 20:09:46.017221 462 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-admin
2026-09-24 20:09:46,284 [I] DB FLAVOR :POSTGRES
2026-09-24 20:09:46,285 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:09:46,285 [I] Checking connection..
2026-09-24 20:09:46,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 20:09:46,518 [I] Checking connection passed.
2026-09-24 20:09:46,518 [I] --------- Verifying version history table ---------
2026-09-24 20:09:46,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:09:46,751 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:09:46,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:09:46,999 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:09:46,999 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 20:09:46,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 20:09:47.252 UTC [12351] WARNING: there is no transaction in progress
2026-09-24 20:09:47,258 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 20:09:47,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:09:47,528 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 20:09:47,528 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:09:47,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 20:09:47,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 20:09:48,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2pm7.c.gcp-upstream.internal','N') ;"
2026-09-24 20:09:48,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:09:48,531 [I] Table x_portal_user does not exist in database ranger
2026-09-24 20:09:48,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:09:48,777 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 20:09:48,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:09:49,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:09:49,255 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 20:09:49,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 20:09:50.196 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.213 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.230 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.246 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.261 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.280 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.335 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.342 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.357 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.369 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.379 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.394 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.405 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.411 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.419 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:50.427 UTC [12459] WARNING: there is no transaction in progress
2026-09-24 20:09:51,088 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 20:09:51,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 20:09:51,349 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 20:09:51,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:09:51,601 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 20:09:51,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:09:51,864 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 20:09:51,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:09:52,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:09:52,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2pm7.c.gcp-upstream.internal';"
2026-09-24 20:09:52,604 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 20:09:52,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 20:09:52,833 [I] DB_PATCHES have already been applied
I20260924 20:09:52.839470 462 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIXC3Dg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 20:09:52.839542 462 mini_ranger.cc:193] Using host: 127.0.115.148
I20260924 20:09:52.843048 462 mini_ranger.cc:227] Ranger admin URL: http://127.0.115.148:36625
Sep 24, 2026 8:09:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:09:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:09:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 8:09:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 8:09:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 8:09:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 8:09:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 20:09:53.973127 462 mini_ranger.cc:160] Time spent starting Ranger: real 8.466s user 0.000s sys 0.010s
Sep 24, 2026 8:10:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 8:10:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 8:10:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 8:10:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 8:10:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 20:10:15.706152 462 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.115.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.115.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/rangerkms_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.115.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/rangerkms_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.115.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.115.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.115.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/HTTP_127.0.115.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 20:10:15 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280615, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 20:10:15.794153 462 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 20:10:15.804464 462 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 20:10:16.676553 462 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 20:10:16.676641 462 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms
I20260924 20:10:16.676663 462 mini_ranger_kms.cc:227] Using postgres at 127.0.115.148:44191
2026-09-24 20:10:16,856 [I] DB FLAVOR :POSTGRES
2026-09-24 20:10:16,857 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:10:16,857 [I] Checking connection
2026-09-24 20:10:16,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 20:10:17,103 [I] connection success
2026-09-24 20:10:17,104 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 20:10:17,104 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 20:10:17,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 20:10:17,348 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 20:10:17,348 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:10:17,348 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 20:10:17,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.115.148:44191/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 20:10:17,634 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 20:10:18.113250 462 mini_ranger_kms.cc:310] Created kms service
I20260924 20:10:18.309037 462 mini_ranger_kms.cc:326] Created kudu user
I20260924 20:10:18.356364 462 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 20:10:18 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280615, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:18.742255 462 mini_ranger_kms.cc:384] Added ranger policy
I20260924 20:10:18.742489 462 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIXC3Dg/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIXC3Dg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 20:10:18.742522 462 mini_ranger_kms.cc:240] Using host: 127.0.115.148
I20260924 20:10:18.746635 462 mini_ranger_kms.cc:276] Ranger KMS PID: 12639
I20260924 20:10:18.746721 462 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.115.148:49641
Sep 24, 2026 8:10:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:10:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:10:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 8:10:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 8:10:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 8:10:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 20:10:21.067698 462 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.274s user 0.003s sys 0.161s
I20260924 20:10:21.067849 462 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 20:10:21.067996 462 mini_ranger_kms.cc:401] 127.0.115.148:49641/kms/v1/keys
Sep 24 20:10:21 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790280621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.115.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 20:10:21 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.115.148@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
Sep 24 20:10:22 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.115.148@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
Sep 24 20:10:24 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:10:24.456776 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44629
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44629
--ranger_config_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:24.594614 12690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:24.594955 12690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:24.595053 12690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:24.599246 12690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:24.599356 12690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:24.599392 12690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:24.599431 12690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:24.599462 12690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:24.605610 12690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.115.148:49641/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44629
--ranger_config_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:24.607008 12690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:24.608341 12690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:10:24.619659 12690 server_base.cc:1034] running on GCE node
W20260924 20:10:24.619814 12696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:24.619597 12695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:24.619611 12698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:24.620615 12690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:24.621953 12690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:24.623117 12690 hybrid_clock.cc:648] HybridClock initialized: now 1790280624623103 us; error 27 us; skew 500 ppm
Sep 24 20:10:24 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:24.626107 12690 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:24.627121 12690 webserver.cc:460] Webserver started at http://127.0.115.190:38487/ using document root <none> and password file <none>
I20260924 20:10:24.627811 12690 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:24.627871 12690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:24.628155 12690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:10:24 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:24.679095 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "c3882efc8fa345b7854d59fde8069d67"
format_stamp: "Formatted at 2026-09-24 20:10:24 on dist-test-slave-2pm7"
server_key: "5aa1629a70b360e53ebb85e4a6b86174"
server_key_iv: "a79b7dcac845f7cddcb057d79571d946"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:24.679745 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "c3882efc8fa345b7854d59fde8069d67"
format_stamp: "Formatted at 2026-09-24 20:10:24 on dist-test-slave-2pm7"
server_key: "5aa1629a70b360e53ebb85e4a6b86174"
server_key_iv: "a79b7dcac845f7cddcb057d79571d946"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:24.684178 12690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:10:24.698866 12705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:24.700621 12690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:10:24.700788 12690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "c3882efc8fa345b7854d59fde8069d67"
format_stamp: "Formatted at 2026-09-24 20:10:24 on dist-test-slave-2pm7"
server_key: "5aa1629a70b360e53ebb85e4a6b86174"
server_key_iv: "a79b7dcac845f7cddcb057d79571d946"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:24.700946 12690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:24.724279 12690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:24.728034 12690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:24.728632 12690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:24.736666 12690 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44629
I20260924 20:10:24.736673 12757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44629 every 8 connection(s)
I20260924 20:10:24.737972 12690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:24.742187 12758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:24.745385 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 12690
I20260924 20:10:24.745517 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:10:24.745886 462 external_mini_cluster.cc:1428] Setting key 708b48b05a994acf1491afce8c924b5e
I20260924 20:10:24.749188 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Bootstrap starting.
I20260924 20:10:24.752087 12758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:24.753059 12758 log.cc:824] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:24.755520 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: No bootstrap required, opened a new log
Sep 24 20:10:24 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:24.759414 12758 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:24.759662 12758 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:24.759735 12758 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3882efc8fa345b7854d59fde8069d67, State: Initialized, Role: FOLLOWER
I20260924 20:10:24.760308 12758 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:24.760468 12758 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:10:24.760555 12758 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:10:24.760723 12758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:24.761013 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:24.747580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.115.190:44629)
0924 20:10:24.748052 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:24.748069 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:24.748105 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:10:24.748716 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:24.748732 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:24.749068 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:24.749417 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:24.750471 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:24.751459 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:24.752703 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:24.753268 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:24.753437 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:24.756287 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:24.756293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:24.756306 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:24.756332 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:24.757985 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:24.758427 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:24.758435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:24.758446 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:24.758568 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:24.758874 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:24.758877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:24.758879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:24.759304 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:24.759559 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:24.759830 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 20:10:24.760152 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":146,"threads_started":1}
I20260924 20:10:24.761700 12758 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:24.762109 12758 leader_election.cc:302] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3882efc8fa345b7854d59fde8069d67; no voters:
I20260924 20:10:24.762472 12758 leader_election.cc:288] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:10:24.762562 12763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:24.762797 12763 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 LEADER]: Becoming Leader. State: Replica: c3882efc8fa345b7854d59fde8069d67, State: Running, Role: LEADER
I20260924 20:10:24.763221 12763 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:24.763757 12758 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:24.765542 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3882efc8fa345b7854d59fde8069d67. Latest consensus state: current_term: 1 leader_uuid: "c3882efc8fa345b7854d59fde8069d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } } }
I20260924 20:10:24.765522 12764 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3882efc8fa345b7854d59fde8069d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } } }
I20260924 20:10:24.765669 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:24.765669 12764 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:24.766215 12773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:24.766672 12758 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:10:24.773782 12773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:24.781329 12773 catalog_manager.cc:1347] Generated new cluster ID: 099812dbf39446b19e8138a819e2659a
I20260924 20:10:24.781417 12773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:10:24.801299 12773 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:10:24.802959 12773 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:10:24.816504 12773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Generated new TSK 0
I20260924 20:10:24.817366 12773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:10:25.575361 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44629
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:25.736733 12803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:25.737154 12803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:25.737278 12803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:25.743470 12803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:25.743628 12803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:25.743755 12803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:10:25.750883 12803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.115.148:49641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:25.752029 12803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:25.753371 12803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:25.765697 12813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:25.765695 12811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:25.766160 12810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:25.766700 12803 server_base.cc:1034] running on GCE node
I20260924 20:10:25.767158 12803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:25.767879 12803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:25.771322 12803 hybrid_clock.cc:648] HybridClock initialized: now 1790280625771273 us; error 55 us; skew 500 ppm
Sep 24 20:10:25 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:25.774413 12803 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:10:25.775390 12803 webserver.cc:460] Webserver started at http://127.0.115.129:32859/ using document root <none> and password file <none>
I20260924 20:10:25.776054 12803 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:25.776145 12803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:25.776366 12803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:10:25 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:25.794060 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "0f80432702f7407699f09045bd2e21c3"
format_stamp: "Formatted at 2026-09-24 20:10:25 on dist-test-slave-2pm7"
server_key: "e760e2aa3c9c5a2db8fbe22d8df94455"
server_key_iv: "2d309c3c5b38d78f9d4b98495365bff4"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:25.794798 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "0f80432702f7407699f09045bd2e21c3"
format_stamp: "Formatted at 2026-09-24 20:10:25 on dist-test-slave-2pm7"
server_key: "e760e2aa3c9c5a2db8fbe22d8df94455"
server_key_iv: "2d309c3c5b38d78f9d4b98495365bff4"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:25.813829 12803 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.006s sys 0.000s
I20260924 20:10:25.836349 12822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:25.838200 12803 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 20:10:25.838349 12803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "0f80432702f7407699f09045bd2e21c3"
format_stamp: "Formatted at 2026-09-24 20:10:25 on dist-test-slave-2pm7"
server_key: "e760e2aa3c9c5a2db8fbe22d8df94455"
server_key_iv: "2d309c3c5b38d78f9d4b98495365bff4"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:25.838495 12803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:25.886209 12803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:25.890027 12803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:25.890291 12803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:25.891109 12803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:25.892326 12803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:25.892432 12803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:25.892560 12803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:25.892635 12803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:25.906424 12803 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:33859
I20260924 20:10:25.907539 12935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:33859 every 8 connection(s)
I20260924 20:10:25.907730 12803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:10:25.908325 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 12803
I20260924 20:10:25.908460 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:10:25.908818 462 external_mini_cluster.cc:1428] Setting key cd4ac88016b6700792d1c807a7d36e7f
Sep 24 20:10:25 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:10:25.932174 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:25.914616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44629 (local address 127.0.115.129:41179)
0924 20:10:25.917559 (+ 2943us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:25.917614 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:25.918614 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:25.919521 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:25.919531 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:25.919882 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:25.920571 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:25.920589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:25.921686 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:25.921690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:25.923117 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:25.923128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:25.923906 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:25.923910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:25.924124 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:25.924754 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:25.924775 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:25.926672 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:25.928930 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:25.928936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:25.928953 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:25.929283 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:25.929595 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:25.929598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:25.929600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:25.929770 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:25.930230 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:25.930237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:25.930557 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 20:10:25.930758 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 20:10:25.931088 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2698,"thread_start_us":163,"threads_started":1}
I20260924 20:10:25.932961 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:25.915751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:41179 (local address 127.0.115.190:44629)
0924 20:10:25.919021 (+ 3270us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:25.919026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:25.919040 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:10:25.919083 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:25.919086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:25.919133 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:25.919218 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:25.920736 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:25.921527 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:25.923256 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:25.923632 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:25.923730 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:25.926861 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:25.926865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:25.926868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:25.926893 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:25.928789 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:25.929394 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:25.929397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:25.929399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:25.929484 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:25.929875 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:25.929878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:25.929880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:25.930092 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:25.930179 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:25.932607 (+ 2428us) server_negotiation.cc:299] Negotiation successful
0924 20:10:25.932734 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3193,"thread_start_us":77,"threads_started":1}
Sep 24 20:10:25 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790280625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:25.935304 12936 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44629
I20260924 20:10:25.935562 12936 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:25.936149 12936 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:25.938059 12722 ts_manager.cc:194] Registered new tserver with Master: 0f80432702f7407699f09045bd2e21c3 (127.0.115.129:33859)
I20260924 20:10:25.939638 12722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:41179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:10:25.977042 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44629
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:26.129037 12942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:26.129437 12942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:26.129519 12942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:26.135918 12942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:26.136022 12942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:26.136116 12942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:10:26.144379 12942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.115.148:49641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:26.146021 12942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:26.147444 12942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:26.160578 12949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:26.160682 12948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:26.161015 12942 server_base.cc:1034] running on GCE node
W20260924 20:10:26.160599 12951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:26.161603 12942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:26.162449 12942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:26.165748 12942 hybrid_clock.cc:648] HybridClock initialized: now 1790280626165659 us; error 99 us; skew 500 ppm
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:26.169476 12942 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:10:26.170634 12942 webserver.cc:460] Webserver started at http://127.0.115.130:34471/ using document root <none> and password file <none>
I20260924 20:10:26.171411 12942 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:26.171504 12942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:26.171729 12942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:26.188055 12942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "536e1bebb4b949efbc9bbc6e761116f1"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "4002889de61baba96068760f8f689aa3"
server_key_iv: "7e993b0c2d0facb2ee4a1126ce659074"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:26.188870 12942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "536e1bebb4b949efbc9bbc6e761116f1"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "4002889de61baba96068760f8f689aa3"
server_key_iv: "7e993b0c2d0facb2ee4a1126ce659074"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:26.197448 12942 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.004s
I20260924 20:10:26.215058 12958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:26.216967 12942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:10:26.217118 12942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "536e1bebb4b949efbc9bbc6e761116f1"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "4002889de61baba96068760f8f689aa3"
server_key_iv: "7e993b0c2d0facb2ee4a1126ce659074"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:26.217273 12942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:26.243359 12942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:26.247139 12942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:26.247418 12942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:26.248205 12942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:26.249279 12942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:26.249394 12942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:26.249495 12942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:26.249557 12942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:26.263769 12942 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:38973
I20260924 20:10:26.265110 12942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:10:26.266144 13071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:38973 every 8 connection(s)
I20260924 20:10:26.273104 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 12942
I20260924 20:10:26.273229 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:10:26.273581 462 external_mini_cluster.cc:1428] Setting key 6a28a2b7cc3181834a425c25a542b089
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:26.283963 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:26.268596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:58921 (local address 127.0.115.190:44629)
0924 20:10:26.268756 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:26.268761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:26.269907 (+ 1146us) server_negotiation.cc:407] Connection header received
0924 20:10:26.271201 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:26.271205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:26.271264 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:26.271322 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:26.273161 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.273994 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.275735 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.276103 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.276183 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:26.279285 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:26.279289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:26.279293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:26.279320 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:26.281336 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:26.282008 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:26.282011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:26.282013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:26.282108 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:26.282527 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:26.282531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:26.282533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:26.282776 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:26.282856 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:26.283596 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 20:10:26.283742 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:10:26.285977 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:26.269135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44629 (local address 127.0.115.130:58921)
0924 20:10:26.269734 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:26.269795 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:26.270931 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:26.271786 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:26.271795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:26.272210 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:26.272960 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:26.272989 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.274162 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.274165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:26.275595 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:26.275606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.276399 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.276403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:26.276607 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:26.277284 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:26.277306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:26.279099 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:26.281481 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:26.281487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:26.281505 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:26.281881 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:26.282226 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:26.282229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:26.282231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:26.282410 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:26.282946 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:26.282952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:26.283279 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 20:10:26.284445 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0924 20:10:26.284808 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":163,"threads_started":1}
I20260924 20:10:26.287561 13072 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44629
I20260924 20:10:26.287822 13072 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:26.288434 13072 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:26.289733 12722 ts_manager.cc:194] Registered new tserver with Master: 536e1bebb4b949efbc9bbc6e761116f1 (127.0.115.130:38973)
I20260924 20:10:26.290494 12722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:58921
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:10:26.324541 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44629
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
W20260924 20:10:26.523788 13077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:26.524678 13077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:26.524784 13077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:26.531059 13077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:26.531224 13077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:26.531358 13077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:10:26.539788 13077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.115.148:49641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:26.541481 13077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:26.542944 13077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:10:26.558899 13077 server_base.cc:1034] running on GCE node
W20260924 20:10:26.559283 13082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:26.559641 13085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:26.568030 13083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:26.568770 13077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:26.569706 13077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:26.601291 13077 hybrid_clock.cc:648] HybridClock initialized: now 1790280626601157 us; error 167 us; skew 500 ppm
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:26.611893 13077 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:10:26.613437 13077 webserver.cc:460] Webserver started at http://127.0.115.131:46683/ using document root <none> and password file <none>
I20260924 20:10:26.614218 13077 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:26.614292 13077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:26.614549 13077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:26.642630 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "42441fec8686467792dd9a936c1b0fe4"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "1152fe1bf835bc1c40822b4354dc2a96"
server_key_iv: "fd309f9cc3ae0e9dd7bf14f80194257f"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:26.648480 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "42441fec8686467792dd9a936c1b0fe4"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "1152fe1bf835bc1c40822b4354dc2a96"
server_key_iv: "fd309f9cc3ae0e9dd7bf14f80194257f"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:26.664741 13077 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.001s sys 0.004s
I20260924 20:10:26.687018 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:26.688951 13077 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 20:10:26.689088 13077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "42441fec8686467792dd9a936c1b0fe4"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "1152fe1bf835bc1c40822b4354dc2a96"
server_key_iv: "fd309f9cc3ae0e9dd7bf14f80194257f"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:26.689230 13077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:26.721401 13077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:26.726464 13077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:26.726766 13077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:26.727591 13077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:26.728858 13077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:26.728967 13077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:26.729069 13077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:26.729135 13077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:26.743445 13077 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:37137
I20260924 20:10:26.744714 13077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:10:26.745321 13205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:37137 every 8 connection(s)
I20260924 20:10:26.748458 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13077
I20260924 20:10:26.748588 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:10:26.748889 462 external_mini_cluster.cc:1428] Setting key 3b78d431d21f96366aa801697ef600bc
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:26.766036 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:26.748144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:34817 (local address 127.0.115.190:44629)
0924 20:10:26.749277 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:26.749282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:26.752496 (+ 3214us) server_negotiation.cc:407] Connection header received
0924 20:10:26.753745 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:26.753749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:26.753805 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:26.753900 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:26.755341 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.756249 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.757937 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.758320 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.758437 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:26.761426 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:26.761430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:26.761433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:26.761463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:26.763252 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:26.764040 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:26.764044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:26.764046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:26.764138 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:26.764572 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:26.764577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:26.764579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:26.764849 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:26.764971 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:26.765702 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 20:10:26.765853 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030}
I20260924 20:10:26.767042 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:26.749799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44629 (local address 127.0.115.131:34817)
0924 20:10:26.752302 (+ 2503us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:26.752377 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:26.753484 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:26.754087 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:26.754097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:26.754454 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:26.755168 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:26.755190 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.756397 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.756401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:26.757790 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:26.757802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:26.758501 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:26.758505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:26.758756 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:26.759432 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:26.759453 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:26.761206 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:26.763441 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:26.763447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:26.763463 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:26.763847 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:26.764257 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:26.764260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:26.764262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:26.764438 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:26.765054 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:26.765061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:26.765385 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 20:10:26.765607 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 20:10:26.766004 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2259,"thread_start_us":154,"threads_started":1}
I20260924 20:10:26.768537 13206 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44629
I20260924 20:10:26.768790 13206 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:26.769356 13206 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:26.770547 12722 ts_manager.cc:194] Registered new tserver with Master: 42441fec8686467792dd9a936c1b0fe4 (127.0.115.131:37137)
I20260924 20:10:26.771340 12722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:34817
I20260924 20:10:26.775166 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:10:26.776531 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 12803
I20260924 20:10:26.784410 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 12942
I20260924 20:10:26.792081 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13077
I20260924 20:10:26.800860 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 12690
I20260924 20:10:26.807129 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44629
--webserver_interface=127.0.115.190
--webserver_port=38487
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44629
--ranger_config_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:26.919754 13209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:26.920097 13209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:26.920142 13209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:26.924185 13209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:26.924242 13209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:26.924261 13209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:26.924278 13209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:26.924292 13209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:26.929772 13209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44629
--ranger_config_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=38487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:26.930763 13209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:26.931746 13209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:26.940287 13215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:26.940258 13214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:26.940331 13217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:26.940389 13209 server_base.cc:1034] running on GCE node
I20260924 20:10:26.940985 13209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:26.942051 13209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:26.943210 13209 hybrid_clock.cc:648] HybridClock initialized: now 1790280626943178 us; error 43 us; skew 500 ppm
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:26.945868 13209 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:26.946666 13209 webserver.cc:460] Webserver started at http://127.0.115.190:38487/ using document root <none> and password file <none>
I20260924 20:10:26.947149 13209 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:26.947212 13209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:10:26 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:27.007093 13209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:10:27.009521 13225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.010933 13209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:10:27.011104 13209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "c3882efc8fa345b7854d59fde8069d67"
format_stamp: "Formatted at 2026-09-24 20:10:24 on dist-test-slave-2pm7"
server_key: "5aa1629a70b360e53ebb85e4a6b86174"
server_key_iv: "a79b7dcac845f7cddcb057d79571d946"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:27.011698 13209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:27.031421 13209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:27.034159 13209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:27.034415 13209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:27.041953 13209 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44629
I20260924 20:10:27.041934 13277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44629 every 8 connection(s)
I20260924 20:10:27.043037 13209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:27.044692 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13209
I20260924 20:10:27.045140 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:33859
--local_ip_for_outbound_sockets=127.0.115.129
--tserver_master_addrs=127.0.115.190:44629
--webserver_port=32859
--webserver_interface=127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:10:27.047894 13278 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:10:27.049542 13278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Bootstrap starting.
I20260924 20:10:27.065828 13278 log.cc:824] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:27.072355 13278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:10:27.072751 13278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Bootstrap complete.
I20260924 20:10:27.077148 13278 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:27.077567 13278 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3882efc8fa345b7854d59fde8069d67, State: Initialized, Role: FOLLOWER
I20260924 20:10:27.078045 13278 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:27.078189 13278 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:10:27.078274 13278 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:10:27.078400 13278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:10:27.080339 13278 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:27.080668 13278 leader_election.cc:302] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3882efc8fa345b7854d59fde8069d67; no voters:
I20260924 20:10:27.081013 13278 leader_election.cc:288] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:10:27.081095 13282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:10:27.081321 13282 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [term 2 LEADER]: Becoming Leader. State: Replica: c3882efc8fa345b7854d59fde8069d67, State: Running, Role: LEADER
I20260924 20:10:27.081668 13282 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } }
I20260924 20:10:27.082281 13278 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:27.083166 13283 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3882efc8fa345b7854d59fde8069d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } } }
I20260924 20:10:27.083297 13283 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:27.083541 13284 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3882efc8fa345b7854d59fde8069d67. Latest consensus state: current_term: 2 leader_uuid: "c3882efc8fa345b7854d59fde8069d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3882efc8fa345b7854d59fde8069d67" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44629 } } }
I20260924 20:10:27.083617 13284 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:27.083660 13290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:27.086941 13290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:27.087445 13278 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:10:27.087612 13290 catalog_manager.cc:1259] Loaded cluster ID: 099812dbf39446b19e8138a819e2659a
I20260924 20:10:27.087661 13290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:10:27.090019 13290 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:10:27.090662 13290 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Loaded TSK: 0
I20260924 20:10:27.091853 13290 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:10:27.190932 13280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:27.191274 13280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:27.191346 13280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:27.197857 13280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:27.197983 13280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:27.198091 13280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:10:27.206187 13280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:33859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=32859
--tserver_master_addrs=127.0.115.190:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:27.207778 13280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:27.209224 13280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:27.221719 13318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:27.221728 13315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:27.222326 13316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:27.223405 13280 server_base.cc:1034] running on GCE node
I20260924 20:10:27.223855 13280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:27.224735 13280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:27.225916 13280 hybrid_clock.cc:648] HybridClock initialized: now 1790280627225879 us; error 56 us; skew 500 ppm
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:27.230697 13280 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:10:27.231840 13280 webserver.cc:460] Webserver started at http://127.0.115.129:32859/ using document root <none> and password file <none>
I20260924 20:10:27.232614 13280 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:27.232726 13280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:27.260329 13280 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:10:27.264755 13326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.266366 13280 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 20:10:27.266499 13280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "0f80432702f7407699f09045bd2e21c3"
format_stamp: "Formatted at 2026-09-24 20:10:25 on dist-test-slave-2pm7"
server_key: "e760e2aa3c9c5a2db8fbe22d8df94455"
server_key_iv: "2d309c3c5b38d78f9d4b98495365bff4"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:27.267086 13280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:27.282008 13280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:27.285810 13280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:27.286013 13280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:27.286733 13280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:27.287700 13280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:27.287750 13280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.287822 13280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:27.287904 13280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.302704 13280 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:33859
I20260924 20:10:27.304137 13280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:10:27.305060 13439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:33859 every 8 connection(s)
I20260924 20:10:27.308538 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13280
I20260924 20:10:27.309056 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:38973
--local_ip_for_outbound_sockets=127.0.115.130
--tserver_master_addrs=127.0.115.190:44629
--webserver_port=34471
--webserver_interface=127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:27.328171 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:27.307588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44629 (local address 127.0.115.129:33211)
0924 20:10:27.310100 (+ 2512us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:27.310158 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:27.311225 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:27.313647 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:27.313656 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:27.314013 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:27.314779 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:27.314804 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.317153 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.317156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:27.318212 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:27.318224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.319290 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.319295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:27.319512 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:27.320235 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:27.320256 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:27.321911 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:27.324574 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:27.324580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:27.324596 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:27.324940 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:27.325352 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:27.325356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:27.325358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:27.325560 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:27.326340 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:27.326347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:27.326672 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 20:10:27.326873 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 20:10:27.327216 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2266,"thread_start_us":148,"threads_started":1}
I20260924 20:10:27.329368 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:27.309170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:33211 (local address 127.0.115.190:44629)
0924 20:10:27.312256 (+ 3086us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:27.312280 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:27.312328 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:10:27.312656 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:27.312673 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:27.313115 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:27.315033 (+ 1918us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:27.315376 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.316993 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.318378 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.319104 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.319919 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:27.322136 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:27.322143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:27.322160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:27.322192 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:27.324430 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.325094 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.325101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.325106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.325246 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.325699 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.325702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.325704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.326203 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:27.328066 (+ 1863us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:27.328222 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 20:10:27.328517 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2914,"thread_start_us":108,"threads_started":1}
I20260924 20:10:27.331777 13440 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44629
I20260924 20:10:27.332069 13440 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:27.332643 13440 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
W20260924 20:10:27.334767 13440 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:44629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 20:10:27.334909 13440 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:27.335089 13440 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:27.336035 13440 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:27.336222 13440 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
W20260924 20:10:27.337193 13440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:10:27.457451 13444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:27.457783 13444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:27.457863 13444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:27.462445 13444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:27.462529 13444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:27.462632 13444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:10:27.468973 13444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:38973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=34471
--tserver_master_addrs=127.0.115.190:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:27.470158 13444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:27.471263 13444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:27.482604 13450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:27.482612 13449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:27.482945 13452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:27.484515 13444 server_base.cc:1034] running on GCE node
I20260924 20:10:27.485148 13444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:27.486022 13444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:27.487617 13444 hybrid_clock.cc:648] HybridClock initialized: now 1790280627487535 us; error 95 us; skew 500 ppm
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:27.491846 13444 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:10:27.493130 13444 webserver.cc:460] Webserver started at http://127.0.115.130:34471/ using document root <none> and password file <none>
I20260924 20:10:27.493908 13444 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:27.494022 13444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:27.525755 13444 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.000s
I20260924 20:10:27.529330 13460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.531154 13444 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:10:27.531288 13444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "536e1bebb4b949efbc9bbc6e761116f1"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "4002889de61baba96068760f8f689aa3"
server_key_iv: "7e993b0c2d0facb2ee4a1126ce659074"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:27.532071 13444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:27.578040 13444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:27.581893 13444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:27.582101 13444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:27.582886 13444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:27.584226 13444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:27.584286 13444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.584347 13444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:27.584379 13444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.595325 13444 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:38973
I20260924 20:10:27.598218 13574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:38973 every 8 connection(s)
I20260924 20:10:27.599123 13444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:10:27.607223 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13444
I20260924 20:10:27.607717 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.115.148:49641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:37137
--local_ip_for_outbound_sockets=127.0.115.131
--tserver_master_addrs=127.0.115.190:44629
--webserver_port=46683
--webserver_interface=127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:27.617638 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:27.601561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:33197 (local address 127.0.115.190:44629)
0924 20:10:27.601705 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:27.601709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:27.602754 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 20:10:27.604057 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:27.604062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:27.604125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:27.604186 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:27.605631 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.606168 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.607723 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.608142 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.608276 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:27.612420 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:27.612427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:27.612433 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:27.612470 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:27.614434 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.615271 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.615278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.615281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.615379 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.615916 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.615923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.615926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.616250 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:27.616336 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:27.617244 (+ 908us) server_negotiation.cc:299] Negotiation successful
0924 20:10:27.617405 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:10:27.619892 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:27.602083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44629 (local address 127.0.115.130:33197)
0924 20:10:27.602592 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:27.602633 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:27.603753 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:27.604644 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:27.604655 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:27.604996 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:27.605481 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:27.605494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.606309 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.606314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:27.607583 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:27.607596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.608332 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.608339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:27.608578 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:27.609650 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:27.609677 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:27.612194 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:27.614592 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:27.614601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:27.614622 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:27.615136 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:27.615512 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:27.615518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:27.615522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:27.615730 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:27.616462 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:27.616473 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:27.616858 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 20:10:27.618211 (+ 1353us) client_negotiation.cc:240] Negotiation successful
0924 20:10:27.618622 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":158,"threads_started":1}
I20260924 20:10:27.621390 13575 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44629
I20260924 20:10:27.621668 13575 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:27.622263 13575 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
W20260924 20:10:27.623883 13575 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:44629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 20:10:27.624020 13575 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:27.624214 13575 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:27.624931 13575 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:27.625100 13575 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
W20260924 20:10:27.625756 13575 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:10:27.745865 13578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:27.746181 13578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:27.746253 13578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:27.752580 13578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:27.752656 13578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:27.752740 13578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:10:27.758630 13578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.115.148:49641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:37137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=46683
--tserver_master_addrs=127.0.115.190:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:27.759776 13578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:27.760830 13578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:27.772137 13584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:27.772185 13586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:27.772154 13583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:27.772768 13578 server_base.cc:1034] running on GCE node
I20260924 20:10:27.773159 13578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:27.773821 13578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:27.775036 13578 hybrid_clock.cc:648] HybridClock initialized: now 1790280627775006 us; error 44 us; skew 500 ppm
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:27.780930 13578 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:10:27.782123 13578 webserver.cc:460] Webserver started at http://127.0.115.131:46683/ using document root <none> and password file <none>
I20260924 20:10:27.782996 13578 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:27.783109 13578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:27.812070 13578 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.002s
I20260924 20:10:27.815253 13594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.816937 13578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:10:27.817080 13578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "42441fec8686467792dd9a936c1b0fe4"
format_stamp: "Formatted at 2026-09-24 20:10:26 on dist-test-slave-2pm7"
server_key: "1152fe1bf835bc1c40822b4354dc2a96"
server_key_iv: "fd309f9cc3ae0e9dd7bf14f80194257f"
server_key_version: "kuduclusterkey@0"
I20260924 20:10:27.817854 13578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:27.841566 13578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:27.844739 13578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:27.844940 13578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:27.845600 13578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:27.846489 13578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:27.846546 13578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.846616 13578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:27.846654 13578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:27.857508 13578 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:37137
I20260924 20:10:27.858776 13578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:10:27.859607 13707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:37137 every 8 connection(s)
I20260924 20:10:27.865226 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13578
Sep 24 20:10:27 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:27.878638 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:27.863344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:56841 (local address 127.0.115.190:44629)
0924 20:10:27.863471 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:27.863476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:27.864087 (+ 611us) server_negotiation.cc:407] Connection header received
0924 20:10:27.865629 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:27.865633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:27.865681 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:27.865794 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:27.867749 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.868633 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.870462 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.870815 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.870891 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:27.873973 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:27.873976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:27.873979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:27.874012 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:27.875849 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.876533 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.876537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.876539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.876630 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.877452 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.877454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.877455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.877661 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:27.877750 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:27.878385 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 20:10:27.878488 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:10:27.879784 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:27.862249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:44629 (local address 127.0.115.131:56841)
0924 20:10:27.863867 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:27.863927 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:27.864791 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:27.866476 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:27.866486 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:27.866862 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:27.867568 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:27.867591 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.868800 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.868804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:27.870328 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:27.870339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.871093 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.871097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:27.871290 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:27.872058 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:27.872079 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:27.873778 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:27.876010 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:27.876016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:27.876034 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:27.876409 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:27.877150 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:27.877153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:27.877155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:27.877343 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:27.877763 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:27.877769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:27.878115 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 20:10:27.878325 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 20:10:27.878698 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"thread_start_us":160,"threads_started":1}
I20260924 20:10:27.880947 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:27.866113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.0.115.190:44629)
0924 20:10:27.866452 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:27.866463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:27.868618 (+ 2155us) server_negotiation.cc:407] Connection header received
0924 20:10:27.868846 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:27.868851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:27.868904 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:27.868963 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:27.870048 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.870895 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.872684 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:27.873063 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:27.873133 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:27.874952 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:27.874957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:27.874962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:27.875009 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:27.876569 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.877076 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.877078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.877079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.877146 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:27.877674 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:27.877678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:27.877680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:27.877865 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:27.878078 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:27.880665 (+ 2587us) server_negotiation.cc:299] Negotiation successful
0924 20:10:27.880789 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":53,"threads_started":1}
I20260924 20:10:27.881134 13708 heartbeater.cc:347] Connected to a master server at 127.0.115.190:44629
I20260924 20:10:27.881400 13708 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:27.882066 13708 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:27.883581 13242 ts_manager.cc:194] Registered new tserver with Master: 42441fec8686467792dd9a936c1b0fe4 (127.0.115.131:37137)
I20260924 20:10:27.884939 13242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:56841
Sep 24 20:10:28 dist-test-slave-2pm7 krb5kdc[12246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790280628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:28.337450 13440 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:28.337781 13440 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:28.338619 13241 ts_manager.cc:194] Registered new tserver with Master: 0f80432702f7407699f09045bd2e21c3 (127.0.115.129:33859)
I20260924 20:10:28.339310 13241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:33211
Sep 24 20:10:28 dist-test-slave-2pm7 krb5kdc[12246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for HTTP/127.0.115.148@KRBTEST.COM
I20260924 20:10:28.626004 13575 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:28.626358 13575 heartbeater.cc:511] Master 127.0.115.190:44629 requested a full tablet report, sending...
I20260924 20:10:28.627316 13241 ts_manager.cc:194] Registered new tserver with Master: 536e1bebb4b949efbc9bbc6e761116f1 (127.0.115.130:38973)
I20260924 20:10:28.628172 13241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:33197
I20260924 20:10:28.628369 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:10:28.645180 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.632422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.0.115.190:44629)
0924 20:10:28.632816 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:28.632821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:28.633208 (+ 387us) server_negotiation.cc:407] Connection header received
0924 20:10:28.633415 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:28.633419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:28.633465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:28.633519 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:28.636999 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.637852 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.639050 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.639395 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.639491 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.641254 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:28.641258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:28.641261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:28.641287 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:28.643112 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:28.643648 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:28.643652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:28.643653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:28.643747 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:28.644110 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:28.644114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:28.644116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:28.644342 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:28.644467 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:28.644847 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 20:10:28.644982 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":103,"threads_started":1}
I20260924 20:10:28.650130 13241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:10:28.653731 13241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:10:28.689260 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.678761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:33859 (local address 127.0.0.1:46412)
0924 20:10:28.682246 (+ 3485us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:28.682301 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:28.682456 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:28.684546 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:28.684552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:28.684577 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:28.684746 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.684754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.686185 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.686188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.687476 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.687487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.688739 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.688742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.688845 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.688893 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 20:10:28.689001 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:10:28.689113 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3292,"thread_start_us":86,"threads_started":1}
I20260924 20:10:28.691892 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.681750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.0.115.131:37137)
0924 20:10:28.684367 (+ 2617us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:28.684374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:28.684719 (+ 345us) server_negotiation.cc:407] Connection header received
0924 20:10:28.684800 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:28.684806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:28.685006 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:28.685149 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:28.686303 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.687528 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.689992 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.690859 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.690971 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.691100 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:28.691668 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 20:10:28.691769 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2510,"thread_start_us":106,"threads_started":1}
I20260924 20:10:28.691560 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.681581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:37137 (local address 127.0.0.1:60768)
0924 20:10:28.684508 (+ 2927us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:28.684523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:28.684620 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:28.685300 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:28.685303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:28.685321 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:28.686170 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.686178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.687661 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.687665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.689857 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.689862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.691164 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.691167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.691269 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.691291 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:10:28.691358 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:10:28.691423 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2862,"thread_start_us":86,"threads_started":1}
I20260924 20:10:28.695051 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.682237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:38973 (local address 127.0.0.1:40370)
0924 20:10:28.689380 (+ 7143us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:28.689393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:28.689482 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:28.690401 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:28.690404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:28.690420 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:28.690495 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.690499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.692111 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.692114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.693178 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.693190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.694671 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.694675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.694763 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.694785 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:10:28.694852 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:10:28.694914 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7082,"thread_start_us":87,"threads_started":1}
I20260924 20:10:28.695503 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.679895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.0.115.129:33859)
0924 20:10:28.683545 (+ 3650us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:28.683553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:28.683579 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:10:28.683657 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:28.683663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:28.683877 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:28.684077 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:28.684900 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.686051 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.687616 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.688438 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.688540 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.688680 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:28.694559 (+ 5879us) server_negotiation.cc:299] Negotiation successful
0924 20:10:28.695362 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3492,"thread_start_us":185,"threads_started":1}
I20260924 20:10:28.696486 13642 tablet_service.cc:1467] Processing CreateTablet for tablet 8271950ed2694ed992d9c019f85be523 (DEFAULT_TABLE table=test-table [id=df45c9ae9d5e42a3a910b4c93228810d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:28.697633 13642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8271950ed2694ed992d9c019f85be523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:28.698369 13374 tablet_service.cc:1467] Processing CreateTablet for tablet 8271950ed2694ed992d9c019f85be523 (DEFAULT_TABLE table=test-table [id=df45c9ae9d5e42a3a910b4c93228810d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:28.699380 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8271950ed2694ed992d9c019f85be523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:28.709597 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.682400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.0.115.130:38973)
0924 20:10:28.684442 (+ 2042us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:28.684449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:28.689833 (+ 5384us) server_negotiation.cc:407] Connection header received
0924 20:10:28.689932 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:28.689937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:28.690116 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:28.690248 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:28.690613 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.691679 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.693363 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.694237 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.694343 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.694478 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:28.709295 (+ 14817us) server_negotiation.cc:299] Negotiation successful
0924 20:10:28.709437 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":103,"threads_started":1}
I20260924 20:10:28.711455 13743 tablet_bootstrap.cc:492] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3: Bootstrap starting.
I20260924 20:10:28.713043 13490 tablet_service.cc:1467] Processing CreateTablet for tablet 8271950ed2694ed992d9c019f85be523 (DEFAULT_TABLE table=test-table [id=df45c9ae9d5e42a3a910b4c93228810d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:28.714179 13490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8271950ed2694ed992d9c019f85be523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:28.714311 13743 tablet_bootstrap.cc:654] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:28.715260 13743 log.cc:824] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:28.716109 13742 tablet_bootstrap.cc:492] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Bootstrap starting.
I20260924 20:10:28.718303 13742 tablet_bootstrap.cc:654] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:28.719208 13742 log.cc:824] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:28.721450 13743 tablet_bootstrap.cc:492] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3: No bootstrap required, opened a new log
I20260924 20:10:28.721658 13743 ts_tablet_manager.cc:1397] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 20:10:28.726241 13743 raft_consensus.cc:348] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.726539 13743 raft_consensus.cc:374] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:28.726604 13743 raft_consensus.cc:729] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f80432702f7407699f09045bd2e21c3, State: Initialized, Role: FOLLOWER
I20260924 20:10:28.727092 13743 consensus_queue.cc:260] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.727893 13440 heartbeater.cc:503] Master 127.0.115.190:44629 was elected leader, sending a full tablet report...
I20260924 20:10:28.728271 13743 ts_tablet_manager.cc:1428] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 20:10:28.730680 13742 tablet_bootstrap.cc:492] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: No bootstrap required, opened a new log
I20260924 20:10:28.730892 13742 ts_tablet_manager.cc:1397] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260924 20:10:28.735728 13742 raft_consensus.cc:348] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.736037 13742 raft_consensus.cc:374] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:28.736091 13742 raft_consensus.cc:729] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42441fec8686467792dd9a936c1b0fe4, State: Initialized, Role: FOLLOWER
I20260924 20:10:28.737039 13742 consensus_queue.cc:260] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.738131 13742 ts_tablet_manager.cc:1428] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260924 20:10:28.738701 13746 tablet_bootstrap.cc:492] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Bootstrap starting.
I20260924 20:10:28.740154 13708 heartbeater.cc:503] Master 127.0.115.190:44629 was elected leader, sending a full tablet report...
I20260924 20:10:28.741243 13746 tablet_bootstrap.cc:654] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:28.742165 13746 log.cc:824] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:28.744441 13746 tablet_bootstrap.cc:492] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: No bootstrap required, opened a new log
I20260924 20:10:28.744647 13746 ts_tablet_manager.cc:1397] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:10:28.749147 13746 raft_consensus.cc:348] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.749431 13746 raft_consensus.cc:374] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:28.749493 13746 raft_consensus.cc:729] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536e1bebb4b949efbc9bbc6e761116f1, State: Initialized, Role: FOLLOWER
I20260924 20:10:28.750010 13746 consensus_queue.cc:260] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.751145 13746 ts_tablet_manager.cc:1428] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:10:28.751492 13575 heartbeater.cc:503] Master 127.0.115.190:44629 was elected leader, sending a full tablet report...
I20260924 20:10:28.789366 13750 raft_consensus.cc:482] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:28.789599 13750 raft_consensus.cc:504] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.792519 13750 leader_election.cc:288] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f80432702f7407699f09045bd2e21c3 (127.0.115.129:33859), 42441fec8686467792dd9a936c1b0fe4 (127.0.115.131:37137)
I20260924 20:10:28.797312 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.791523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:44683 (local address 127.0.115.129:33859)
0924 20:10:28.791648 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:28.791653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:28.792857 (+ 1204us) server_negotiation.cc:407] Connection header received
0924 20:10:28.793084 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:28.793087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:28.793134 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:28.793180 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:28.793610 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.794390 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.796004 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.796599 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.796692 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.796747 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:28.797096 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 20:10:28.797175 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:10:28.797735 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.791140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:33859 (local address 127.0.115.130:44683)
0924 20:10:28.792756 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:28.792771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:28.792943 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:28.793314 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:28.793317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:28.793348 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:28.793487 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.793493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.794535 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.794539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.795834 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.795845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.796838 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.796841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.796945 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.796982 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:10:28.797525 (+ 543us) client_negotiation.cc:240] Negotiation successful
0924 20:10:28.797604 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"thread_start_us":110,"threads_started":1}
I20260924 20:10:28.798398 13394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8271950ed2694ed992d9c019f85be523" candidate_uuid: "536e1bebb4b949efbc9bbc6e761116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f80432702f7407699f09045bd2e21c3" is_pre_election: true
I20260924 20:10:28.798789 13394 raft_consensus.cc:2454] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 536e1bebb4b949efbc9bbc6e761116f1 in term 0.
I20260924 20:10:28.799360 13462 leader_election.cc:302] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f80432702f7407699f09045bd2e21c3, 536e1bebb4b949efbc9bbc6e761116f1; no voters:
I20260924 20:10:28.799679 13750 raft_consensus.cc:2781] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:10:28.799763 13750 raft_consensus.cc:482] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:10:28.799808 13750 raft_consensus.cc:3037] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:28.801237 13750 raft_consensus.cc:504] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.801980 13394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8271950ed2694ed992d9c019f85be523" candidate_uuid: "536e1bebb4b949efbc9bbc6e761116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f80432702f7407699f09045bd2e21c3"
I20260924 20:10:28.802109 13394 raft_consensus.cc:3037] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:28.803367 13394 raft_consensus.cc:2454] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536e1bebb4b949efbc9bbc6e761116f1 in term 1.
I20260924 20:10:28.803681 13462 leader_election.cc:302] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f80432702f7407699f09045bd2e21c3, 536e1bebb4b949efbc9bbc6e761116f1; no voters:
I20260924 20:10:28.804121 13750 leader_election.cc:288] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0f80432702f7407699f09045bd2e21c3 (127.0.115.129:33859), 42441fec8686467792dd9a936c1b0fe4 (127.0.115.131:37137)
I20260924 20:10:28.804222 13750 raft_consensus.cc:2781] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:28.804293 13750 raft_consensus.cc:686] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 1 LEADER]: Becoming Leader. State: Replica: 536e1bebb4b949efbc9bbc6e761116f1, State: Running, Role: LEADER
I20260924 20:10:28.804659 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.792129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35963 (local address 127.0.115.131:37137)
0924 20:10:28.792242 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:28.792246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:28.800228 (+ 7982us) server_negotiation.cc:407] Connection header received
0924 20:10:28.800438 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:28.800442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:28.800494 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:28.800550 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:28.801045 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.801806 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.803315 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.803929 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.804044 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.804096 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:28.804450 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 20:10:28.804531 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:10:28.804642 13750 consensus_queue.cc:237] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } }
I20260924 20:10:28.805107 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.792291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:37137 (local address 127.0.115.130:35963)
0924 20:10:28.800122 (+ 7831us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:28.800138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:28.800311 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:28.800715 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:28.800719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:28.800740 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:28.800885 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.800892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.801967 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.801970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.803170 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:28.803182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.804193 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.804198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:28.804307 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.804330 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:10:28.804885 (+ 555us) client_negotiation.cc:240] Negotiation successful
0924 20:10:28.804966 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7755,"thread_start_us":103,"threads_started":1}
I20260924 20:10:28.805791 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8271950ed2694ed992d9c019f85be523" candidate_uuid: "536e1bebb4b949efbc9bbc6e761116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42441fec8686467792dd9a936c1b0fe4" is_pre_election: true
I20260924 20:10:28.806099 13662 raft_consensus.cc:2454] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 536e1bebb4b949efbc9bbc6e761116f1 in term 0.
I20260924 20:10:28.806450 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8271950ed2694ed992d9c019f85be523" candidate_uuid: "536e1bebb4b949efbc9bbc6e761116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42441fec8686467792dd9a936c1b0fe4"
I20260924 20:10:28.806588 13661 raft_consensus.cc:3037] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 0 FOLLOWER]: Advancing to term 1
W20260924 20:10:28.807310 13441 tablet.cc:2369] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:28.807905 13661 raft_consensus.cc:2454] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536e1bebb4b949efbc9bbc6e761116f1 in term 1.
I20260924 20:10:28.810207 13241 catalog_manager.cc:5495] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 536e1bebb4b949efbc9bbc6e761116f1 (127.0.115.130). New cstate: current_term: 1 leader_uuid: "536e1bebb4b949efbc9bbc6e761116f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "536e1bebb4b949efbc9bbc6e761116f1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38973 } health_report { overall_health: HEALTHY } } }
I20260924 20:10:28.847425 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.842481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.0.115.130:38973)
0924 20:10:28.842604 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:28.842608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:28.842621 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:10:28.842711 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:28.842714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:28.842777 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:28.842879 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:10:28.843241 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.844059 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.845263 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:28.845587 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:28.845676 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:28.846076 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:10:28.846656 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:10:28.846769 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:28.847184 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 20:10:28.847246 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:10:28.850778 13576 tablet.cc:2369] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:28.855693 13394 raft_consensus.cc:1261] T 8271950ed2694ed992d9c019f85be523 P 0f80432702f7407699f09045bd2e21c3 [term 1 FOLLOWER]: Refusing update from remote peer 536e1bebb4b949efbc9bbc6e761116f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:28.855693 13661 raft_consensus.cc:1261] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 1 FOLLOWER]: Refusing update from remote peer 536e1bebb4b949efbc9bbc6e761116f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:28.856515 13753 consensus_queue.cc:1035] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f80432702f7407699f09045bd2e21c3" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:28.856722 13750 consensus_queue.cc:1035] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42441fec8686467792dd9a936c1b0fe4" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 37137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:10:28.861510 13709 tablet.cc:2369] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:28.875383 13756 mvcc.cc:205] Tried to move back new op lower bound from 7332989455782785024 to 7332989455586865152. Current Snapshot: MvccSnapshot[applied={T|T < 7332989455782785024}]
I20260924 20:10:28.878747 13758 mvcc.cc:205] Tried to move back new op lower bound from 7332989455782785024 to 7332989455586865152. Current Snapshot: MvccSnapshot[applied={T|T < 7332989455782785024}]
I20260924 20:10:28.881443 13757 mvcc.cc:205] Tried to move back new op lower bound from 7332989455782785024 to 7332989455586865152. Current Snapshot: MvccSnapshot[applied={T|T < 7332989455782785024}]
I20260924 20:10:28.886409 13242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:28.886615 13242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:28.889858 13242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3882efc8fa345b7854d59fde8069d67: Sending DeleteTablet for 3 replicas of tablet 8271950ed2694ed992d9c019f85be523
I20260924 20:10:28.890899 13642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8271950ed2694ed992d9c019f85be523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:28 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:60768
I20260924 20:10:28.892104 13374 tablet_service.cc:1514] Processing DeleteTablet for tablet 8271950ed2694ed992d9c019f85be523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:28 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:46412
I20260924 20:10:28.892715 13490 tablet_service.cc:1514] Processing DeleteTablet for tablet 8271950ed2694ed992d9c019f85be523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:28 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:40370
I20260924 20:10:28.893999 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13280
I20260924 20:10:28.902688 13765 tablet_replica.cc:330] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: stopping tablet replica
I20260924 20:10:28.903051 13767 tablet_replica.cc:330] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: stopping tablet replica
I20260924 20:10:28.903628 13767 raft_consensus.cc:2229] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:10:28.904066 13765 raft_consensus.cc:2229] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:10:28.904548 13765 raft_consensus.cc:2258] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:28.905000 13767 raft_consensus.cc:2258] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:28.906495 13767 ts_tablet_manager.cc:1905] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:28.906495 13765 ts_tablet_manager.cc:1905] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:28.909384 13767 ts_tablet_manager.cc:1918] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:28.909487 13767 log.cc:1196] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/8271950ed2694ed992d9c019f85be523
I20260924 20:10:28.909684 13765 ts_tablet_manager.cc:1918] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:28.909797 13767 ts_tablet_manager.cc:1939] T 8271950ed2694ed992d9c019f85be523 P 536e1bebb4b949efbc9bbc6e761116f1: Deleting consensus metadata
I20260924 20:10:28.909904 13765 log.cc:1196] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/8271950ed2694ed992d9c019f85be523
I20260924 20:10:28.910284 13765 ts_tablet_manager.cc:1939] T 8271950ed2694ed992d9c019f85be523 P 42441fec8686467792dd9a936c1b0fe4: Deleting consensus metadata
I20260924 20:10:28.911104 13227 catalog_manager.cc:4841] TS 536e1bebb4b949efbc9bbc6e761116f1 (127.0.115.130:38973): tablet 8271950ed2694ed992d9c019f85be523 (table test-table [id=df45c9ae9d5e42a3a910b4c93228810d]) successfully deleted
I20260924 20:10:28.911645 13228 catalog_manager.cc:4841] TS 42441fec8686467792dd9a936c1b0fe4 (127.0.115.131:37137): tablet 8271950ed2694ed992d9c019f85be523 (table test-table [id=df45c9ae9d5e42a3a910b4c93228810d]) successfully deleted
W20260924 20:10:28.924490 13227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:33859 (error 108)
I20260924 20:10:28.932135 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13444
I20260924 20:10:28.936435 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:28.935935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:33859 (local address 127.0.0.1:46416)
0924 20:10:28.936234 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:28.936318 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260924 20:10:28.936664 13227 catalog_manager.cc:4584] TS 0f80432702f7407699f09045bd2e21c3 (127.0.115.129:33859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8271950ed2694ed992d9c019f85be523: Network error: Client connection negotiation failed: client connection to 127.0.115.129:33859: connect: Connection refused (error 111)
I20260924 20:10:28.944046 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13578
I20260924 20:10:28.953574 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13209
I20260924 20:10:28.960747 462 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 20:10:29.305838 462 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 20:10:29.305958 462 mini_ranger.cc:66] Stopping Ranger...
I20260924 20:10:29.651146 462 mini_ranger.cc:68] Stopped Ranger
2026-09-24 20:10:29.651 UTC [12276] LOG: received smart shutdown request
2026-09-24 20:10:29.655 UTC [12276] LOG: background worker "logical replication launcher" (PID 12285) exited with exit code 1
2026-09-24 20:10:29.655 UTC [12280] LOG: shutting down
2026-09-24 20:10:29.832 UTC [12276] LOG: database system is shut down
2026-09-24T20:10:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49193 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:10:30 dist-test-slave-2pm7 krb5kdc[13777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:10:30 dist-test-slave-2pm7 krb5kdc[13777](info): set up 2 sockets
Sep 24 20:10:30 dist-test-slave-2pm7 krb5kdc[13777](info): commencing operation
krb5kdc: starting...
W20260924 20:10:30.984277 462 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:10:31.112022 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:37855
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:37855,127.0.115.189:40741,127.0.115.188:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:31.228248 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:31.228564 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:31.228649 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:31.232707 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:31.232787 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:31.232825 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:31.232883 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:31.232921 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:31.238118 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:37855,127.0.115.189:40741,127.0.115.188:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:31.239203 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:31.240317 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:31.248648 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:31.248651 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:31.248780 13790 server_base.cc:1034] running on GCE node
W20260924 20:10:31.248970 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:31.249500 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:31.250917 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:31.252168 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790280631252138 us; error 47 us; skew 500 ppm
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:31.255229 13790 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:31.256171 13790 webserver.cc:460] Webserver started at http://127.0.115.190:36251/ using document root <none> and password file <none>
I20260924 20:10:31.256731 13790 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:31.256822 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:31.257055 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:31.259141 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "aeadcb73ada14b548f21c9ea63fb7a3c"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "584dc9d557d8bca439f3de5bc5e49581"
server_key_iv: "2f4fc5340387cba135c5605e33cc52b4"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.259729 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "aeadcb73ada14b548f21c9ea63fb7a3c"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "584dc9d557d8bca439f3de5bc5e49581"
server_key_iv: "2f4fc5340387cba135c5605e33cc52b4"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.263582 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:10:31.266211 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:31.267473 13790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:10:31.267619 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "aeadcb73ada14b548f21c9ea63fb7a3c"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "584dc9d557d8bca439f3de5bc5e49581"
server_key_iv: "2f4fc5340387cba135c5605e33cc52b4"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.267735 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:31.293416 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:31.297030 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:31.297240 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:31.305313 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:37855
I20260924 20:10:31.305398 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:37855 every 8 connection(s)
I20260924 20:10:31.306367 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:31.308800 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13790
I20260924 20:10:31.308919 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:10:31.309270 462 external_mini_cluster.cc:1428] Setting key 7267e3ff7df2968e13d9f471efcebfab
I20260924 20:10:31.309587 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:31.315593 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.115.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.189@KRBTEST.COM" created.
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:31.332448 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.318689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:37855 (local address 127.0.0.1:51184)
0924 20:10:31.319361 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.319401 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.320313 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.321177 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.321189 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.321633 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.322051 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.322077 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.323244 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.323248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.324093 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.324108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.324782 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.324789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.324946 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.325662 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.325690 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.327329 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.329325 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.329335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.329344 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.329569 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.329895 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.329898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.329900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.330011 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.330674 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.330680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.330820 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.331012 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.331421 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":105,"thread_start_us":184,"threads_started":1}
I20260924 20:10:31.332448 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.318689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.0.115.190:37855)
0924 20:10:31.319289 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.319308 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.320599 (+ 1291us) server_negotiation.cc:407] Connection header received
0924 20:10:31.320750 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.320754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.320939 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.321130 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.322231 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.323111 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.324228 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.324612 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.324734 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.327486 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.327492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.327502 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.327521 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.329196 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.329680 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.329686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.329688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.329793 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.330108 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.330111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.330113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.330552 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.330691 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.331076 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.331341 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"spinlock_wait_cycles":102272,"thread_start_us":187,"threads_started":1}
I20260924 20:10:31.334959 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:10:31.335758 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.335465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36546)
0924 20:10:31.335580 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.335646 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:10:31.335996 13806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
I20260924 20:10:31.336789 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.336609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36558)
0924 20:10:31.336697 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.336739 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 20:10:31.336936 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:40741: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab.
Entry for principal kudu/127.0.115.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab.
I20260924 20:10:31.338814 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:40741
--webserver_interface=127.0.115.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab
--principal=kudu/127.0.115.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:37855,127.0.115.189:40741,127.0.115.188:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:10:31.386281 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } attempt: 1
I20260924 20:10:31.387272 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.386915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36562)
0924 20:10:31.387099 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.387174 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 20:10:31.388131 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.387875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36574)
0924 20:10:31.388007 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.388061 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 20:10:31.388320 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:40741: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
I20260924 20:10:31.456524 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } attempt: 2
W20260924 20:10:31.456645 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:31.456976 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:31.457072 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:10:31.457453 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.457118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36584)
0924 20:10:31.457287 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.457354 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 20:10:31.458158 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.457934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36594)
0924 20:10:31.458044 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.458093 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:10:31.458287 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:40741: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
W20260924 20:10:31.461611 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:31.461663 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:31.461681 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:31.461723 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:31.461756 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:31.467196 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:37855,127.0.115.189:40741,127.0.115.188:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab
--principal=kudu/127.0.115.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:40741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.115.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:31.468328 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:31.469363 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:31.478235 13873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:31.478193 13871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:31.478243 13865 server_base.cc:1034] running on GCE node
W20260924 20:10:31.478197 13870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:31.479036 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:31.480243 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:31.481510 13865 hybrid_clock.cc:648] HybridClock initialized: now 1790280631481481 us; error 38 us; skew 500 ppm
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:31.485706 13865 init.cc:377] Logged in from keytab as kudu/127.0.115.189@KRBTEST.COM (short username kudu)
I20260924 20:10:31.486873 13865 webserver.cc:460] Webserver started at http://127.0.115.189:37693/ using document root <none> and password file <none>
I20260924 20:10:31.487630 13865 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:31.487737 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:31.488004 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:31.490653 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data/instance:
uuid: "13b53b1789304b2881ad4a33bb9b88d8"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "11ede0a963687893dcfe948fb87e322e"
server_key_iv: "8f1fea34ed67b14657dd30703072c7dc"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.491317 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/wal/instance:
uuid: "13b53b1789304b2881ad4a33bb9b88d8"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "11ede0a963687893dcfe948fb87e322e"
server_key_iv: "8f1fea34ed67b14657dd30703072c7dc"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.495146 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 20:10:31.497659 13880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:31.498903 13865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:10:31.499054 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/wal
uuid: "13b53b1789304b2881ad4a33bb9b88d8"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "11ede0a963687893dcfe948fb87e322e"
server_key_iv: "8f1fea34ed67b14657dd30703072c7dc"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.499166 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:31.540949 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:31.544070 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:31.544277 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:31.549446 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } attempt: 3
I20260924 20:10:31.550375 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.550013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36606)
0924 20:10:31.550184 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.550267 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 20:10:31.551126 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.550860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36618)
0924 20:10:31.550993 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.551051 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 20:10:31.551294 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:40741: Network error: Client connection negotiation failed: client connection to 127.0.115.189:40741: connect: Connection refused (error 111)
I20260924 20:10:31.553023 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.189:40741
I20260924 20:10:31.553049 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.189:40741 every 8 connection(s)
I20260924 20:10:31.554066 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
I20260924 20:10:31.555662 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13865
I20260924 20:10:31.555790 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-1/wal/instance
I20260924 20:10:31.556140 462 external_mini_cluster.cc:1428] Setting key 3bc7ca83494252b9f6d4bea592541804
I20260924 20:10:31.557560 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:31.561916 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.115.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.188@KRBTEST.COM" created.
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:31.577548 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.564018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.0.115.190:37855)
0924 20:10:31.564185 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.564188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.565167 (+ 979us) server_negotiation.cc:407] Connection header received
0924 20:10:31.566190 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.566193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.566255 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.566361 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.568050 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.568583 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.570508 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.570753 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.570845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.573732 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.573736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.573739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.573770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.575430 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.575988 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.575992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.575994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.576079 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.576400 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.576403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.576404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.576647 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.576729 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.577266 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.577390 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:10:31.578474 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.564436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:37855 (local address 127.0.0.1:51194)
0924 20:10:31.564997 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.565058 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.565974 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.566596 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.566603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.567059 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.567818 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.567842 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.568722 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.568727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.570298 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.570312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.570938 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.570945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.571219 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.571980 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.572002 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.573549 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.575570 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.575575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.575589 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.575841 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.576169 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.576171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.576173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.576309 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.576745 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.576751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.577002 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.577201 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.577523 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":125,"threads_started":1}
I20260924 20:10:31.580076 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab.
Entry for principal kudu/127.0.115.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab.
I20260924 20:10:31.583544 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:33335
--webserver_interface=127.0.115.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab
--principal=kudu/127.0.115.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:37855,127.0.115.189:40741,127.0.115.188:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
I20260924 20:10:31.590066 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.580570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36628)
0924 20:10:31.580687 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.580703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.580777 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.581431 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.581433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.581561 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.581646 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.581651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.582679 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.582686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.583335 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.583344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.584289 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.584292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.584379 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.584937 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.584970 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.586159 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.588470 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.588472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.588474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.588668 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.589001 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.589003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.589004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.589075 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.589676 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.589678 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.589757 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.589838 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.589938 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:10:31.590436 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.580739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.0.115.189:40741)
0924 20:10:31.581101 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.581110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.581129 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:10:31.581164 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.581168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.581313 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.581419 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.581743 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.582559 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.583466 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.584162 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.584286 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.586297 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.586310 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.586327 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.586366 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.588354 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.588780 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.588790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.588794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.588897 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.589167 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.589173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.589177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.589560 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.589718 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.589982 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.590153 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":143,"threads_started":1}
I20260924 20:10:31.592324 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:10:31.593027 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.592777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56756)
0924 20:10:31.592891 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.592976 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 20:10:31.593330 13882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
I20260924 20:10:31.594283 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.594089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56770)
0924 20:10:31.594196 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.594239 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:10:31.594515 13933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:33335: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
I20260924 20:10:31.643827 13933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } attempt: 1
I20260924 20:10:31.644964 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.644627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56782)
0924 20:10:31.644794 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.644869 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 20:10:31.645750 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.645553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56788)
0924 20:10:31.645654 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.645699 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 20:10:31.645896 13933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:33335: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
I20260924 20:10:31.694458 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } attempt: 4
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
W20260924 20:10:31.705230 13939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:31.705506 13939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:31.705613 13939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:10:31.708056 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.695068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36644)
0924 20:10:31.695316 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.695330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.695421 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.695720 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.695722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.695884 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.696473 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.696483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.698043 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.698046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.698619 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.698627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.699226 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.699229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.699307 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.703098 (+ 3791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.703128 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.704537 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.706273 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.706277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.706279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.706603 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.706942 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.706944 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.706946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.707035 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.707504 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.707508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.707612 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.707716 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.707850 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 20:10:31.708169 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.695207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.0.115.189:40741)
0924 20:10:31.695382 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.695387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.695406 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:10:31.695538 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.695541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.695598 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.695689 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.696614 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.697920 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.698740 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.699103 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.699210 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.704692 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.704695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.704697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.704720 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.706165 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.706736 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.706740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.706741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.706837 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.707129 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.707132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.707133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.707407 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.707497 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.707837 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.708017 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:10:31.708987 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:10:31.709741 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.709467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56794)
0924 20:10:31.709626 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.709675 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 20:10:31.710438 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.710210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56804)
0924 20:10:31.710333 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.710385 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 20:10:31.710602 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:33335: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
W20260924 20:10:31.711040 13939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:31.711110 13939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:31.711154 13939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:31.711190 13939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:31.711226 13939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:31.714073 13933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } attempt: 2
I20260924 20:10:31.714638 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.714425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56820)
0924 20:10:31.714536 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.714583 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 20:10:31.715298 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.715095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56822)
0924 20:10:31.715198 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.715246 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:10:31.715415 13933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:33335: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
I20260924 20:10:31.716980 13939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:37855,127.0.115.189:40741,127.0.115.188:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab
--principal=kudu/127.0.115.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.115.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:31.718081 13939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:31.719084 13939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:31.727672 13948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:31.727690 13939 server_base.cc:1034] running on GCE node
W20260924 20:10:31.727746 13946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:31.727897 13945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:31.728483 13939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:31.729665 13939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:31.730858 13939 hybrid_clock.cc:648] HybridClock initialized: now 1790280631730849 us; error 36 us; skew 500 ppm
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:31.733886 13939 init.cc:377] Logged in from keytab as kudu/127.0.115.188@KRBTEST.COM (short username kudu)
I20260924 20:10:31.734751 13939 webserver.cc:460] Webserver started at http://127.0.115.188:43907/ using document root <none> and password file <none>
I20260924 20:10:31.735322 13939 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:31.735412 13939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:31.735648 13939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:31.737730 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data/instance:
uuid: "bad5891e2739478387220cd56be68790"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "8bb35635a1869fb9ac62fd6889fd3f07"
server_key_iv: "8ee8cd645b51db3a7f1e1271509b0da1"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.738301 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/wal/instance:
uuid: "bad5891e2739478387220cd56be68790"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "8bb35635a1869fb9ac62fd6889fd3f07"
server_key_iv: "8ee8cd645b51db3a7f1e1271509b0da1"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.742017 13939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:10:31.744433 13955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:31.745616 13939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:31.745767 13939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/wal
uuid: "bad5891e2739478387220cd56be68790"
format_stamp: "Formatted at 2026-09-24 20:10:31 on dist-test-slave-2pm7"
server_key: "8bb35635a1869fb9ac62fd6889fd3f07"
server_key_iv: "8ee8cd645b51db3a7f1e1271509b0da1"
server_key_version: "encryptionkey@0"
I20260924 20:10:31.745878 13939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:31.766234 13939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:31.769234 13939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:31.769446 13939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:31.769747 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } attempt: 1
I20260924 20:10:31.770663 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.770262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56830)
0924 20:10:31.770482 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.770566 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 20:10:31.771592 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.771340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56834)
0924 20:10:31.771451 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.771524 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 20:10:31.771724 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:33335: Network error: Client connection negotiation failed: client connection to 127.0.115.188:33335: connect: Connection refused (error 111)
I20260924 20:10:31.777495 13939 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.188:33335
I20260924 20:10:31.777576 14007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.188:33335 every 8 connection(s)
I20260924 20:10:31.778486 13939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
I20260924 20:10:31.780719 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 13939
I20260924 20:10:31.780851 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790280466534371-462-0/minicluster-data/master-2/wal/instance
I20260924 20:10:31.781148 462 external_mini_cluster.cc:1428] Setting key a1997c1f8bacb5938648d742a3d7152d
I20260924 20:10:31.781503 14008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:31.786062 14008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:31.797461 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.782555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.0.115.188:33335)
0924 20:10:31.783013 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.783027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.784281 (+ 1254us) server_negotiation.cc:407] Connection header received
0924 20:10:31.785176 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.785200 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.785624 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.786002 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.786665 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.788097 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.789261 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.789850 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.790009 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.792572 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.792580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.792596 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.792619 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.794294 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.794764 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.794769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.794775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.794884 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.795164 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.795169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.795170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.795561 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.795790 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.796082 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.796399 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260924 20:10:31.797623 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.788098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:37855 (local address 127.0.0.1:51210)
0924 20:10:31.788535 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.788589 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.788706 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.789005 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.789011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.789445 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.789564 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.789570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.790489 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.790494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.791111 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.791119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.791730 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.791733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.791814 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.792406 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.792422 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.794139 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.795880 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.795887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.795892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.796178 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.796540 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.796543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.796544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.796619 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.797050 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.797055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.797147 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.797254 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.797380 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":156,"threads_started":1}
I20260924 20:10:31.797642 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.788133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.0.115.190:37855)
0924 20:10:31.788284 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.788289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.788677 (+ 388us) server_negotiation.cc:407] Connection header received
0924 20:10:31.788805 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.788808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.788862 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.788932 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.789692 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.790358 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.791235 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.791576 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.791686 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.794286 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.794289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.794291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.794315 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.795745 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.796340 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.796346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.796350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.796443 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.796716 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.796720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.796722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.796945 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.797043 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.797368 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.797493 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:10:31.798851 14008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:31.806550 13933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } attempt: 3
I20260924 20:10:31.808156 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.799403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.0.115.189:40741)
0924 20:10:31.799557 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.799564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.799577 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:10:31.799617 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.799620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.799702 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.799795 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.800682 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.801186 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.801938 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.802174 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.802282 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.804617 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.804622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.804628 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.804663 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.806112 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.806648 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.806654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.806658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.806754 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.807095 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.807099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.807100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.807351 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.807466 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.807827 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.807997 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:10:31.808564 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.799284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:40741 (local address 127.0.0.1:36652)
0924 20:10:31.799392 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.799404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.799494 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.799809 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.799812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.800008 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.800578 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.800588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.801292 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.801299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.801812 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.801819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.802282 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.802285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.802380 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.802848 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.802862 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.804453 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.806239 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.806243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.806247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.806525 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.806879 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.806882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.806883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.806973 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.807520 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.807524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.807623 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.808277 (+ 654us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.808400 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 20:10:31.809325 14008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:10:31.809517 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.800159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.0.115.190:37855)
0924 20:10:31.800281 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.800285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.800300 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:10:31.800417 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.800421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.800472 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.800561 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.801222 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.801695 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.802547 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.802795 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.802905 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.805791 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.805794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.805797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.805820 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.807426 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.807889 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.807892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.807894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.808010 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.808337 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.808340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.808342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.808580 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.808843 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.809270 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.809380 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
I20260924 20:10:31.815923 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.807187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.0.115.188:33335)
0924 20:10:31.807311 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.807315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.807395 (+ 80us) server_negotiation.cc:407] Connection header received
0924 20:10:31.807499 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.807502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.807538 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.807629 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.808048 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.808788 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.809793 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.810143 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.810606 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.812359 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.812363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.812365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.812389 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.814057 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.814537 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.814541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.814542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.814625 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.815014 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.815017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.815019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.815240 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.815348 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.815630 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.815776 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":3712}
I20260924 20:10:31.816169 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.806986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56846)
0924 20:10:31.807336 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.807349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.807428 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.807642 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.807645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.807807 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.807926 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.807936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.808893 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.808896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.809690 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.809701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.810273 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.810278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.810381 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.810796 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.810807 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.812201 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.814174 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.814180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.814185 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.814415 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.814751 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.814755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.814757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.814885 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.815358 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.815362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.815458 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.815585 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.815718 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"spinlock_wait_cycles":50944}
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
I20260924 20:10:31.819279 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.809793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56848)
0924 20:10:31.809887 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.809900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.809959 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.810806 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.810809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.810972 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.811078 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.811085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.812114 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.812116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.812786 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.812799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.813483 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.813489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.813582 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.814016 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.814040 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.815410 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.817522 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.817525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.817527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.817759 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.818085 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.818088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.818089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.818208 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.818747 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.818750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.818843 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.818967 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.819107 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"spinlock_wait_cycles":1920}
I20260924 20:10:31.819444 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.809819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.0.115.188:33335)
0924 20:10:31.810352 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.810362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.810376 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:10:31.810417 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.810420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.810460 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.810548 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.811199 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.811993 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.812905 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.813273 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.813425 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.815547 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.815552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.815557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.815584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.817415 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.817869 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.817872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.817874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.817955 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.818316 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.818342 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.818347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.818629 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.818727 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.819033 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.819212 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
I20260924 20:10:31.820737 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.811277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.0.115.189:40741)
0924 20:10:31.811389 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.811393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.811406 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:10:31.811442 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.811448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.811493 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.811569 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.812297 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.813025 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.814234 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.814610 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.814710 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.817629 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.817631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.817633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.817653 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.819186 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.819639 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.819643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.819644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.819725 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.820002 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.820005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.820007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.820196 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.820403 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.820508 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.820610 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:10:31.820824 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8: Bootstrap starting.
I20260924 20:10:31.823191 13933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:31.824313 13933 log.cc:824] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:31.826611 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8: No bootstrap required, opened a new log
I20260924 20:10:31.826934 14008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: Bootstrap starting.
I20260924 20:10:31.827641 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:10:31.829674 14008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:31.830090 13933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:31.830318 13933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:31.830394 13933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b53b1789304b2881ad4a33bb9b88d8, State: Initialized, Role: FOLLOWER
I20260924 20:10:31.830564 14008 log.cc:824] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:31.830894 13933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:31.831390 14023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:31.831605 14023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:10:31.832366 13933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:31.835253 14008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: No bootstrap required, opened a new log
I20260924 20:10:31.839423 14008 raft_consensus.cc:348] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:31.839704 14008 raft_consensus.cc:374] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:31.839769 14008 raft_consensus.cc:729] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad5891e2739478387220cd56be68790, State: Initialized, Role: FOLLOWER
I20260924 20:10:31.840374 14008 consensus_queue.cc:260] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:31.841174 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.830063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.0.115.190:37855)
0924 20:10:31.830349 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.830354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.831137 (+ 783us) server_negotiation.cc:407] Connection header received
0924 20:10:31.831317 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.831320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.831371 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.831431 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.832343 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.833102 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.834344 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.834671 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.834777 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.836662 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.836666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.836669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.836695 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.838285 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.838853 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.838857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.838859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.838964 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.839281 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.839284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.839286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.840260 (+ 974us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.840365 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.840892 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.841028 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260924 20:10:31.841929 14008 sys_catalog.cc:564] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:31.842466 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } attempt: 2
I20260924 20:10:31.843097 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.830607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.0.115.188:33335)
0924 20:10:31.831007 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.831012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.832531 (+ 1519us) server_negotiation.cc:407] Connection header received
0924 20:10:31.832678 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.832681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.832724 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.832769 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.833673 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.834424 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.835452 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.835811 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.835920 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.839087 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.839092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.839095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.839121 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.840868 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.841435 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.841438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.841440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.841528 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.841812 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.841815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.841817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.842027 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.842234 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.842832 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.842949 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260924 20:10:31.838811 13865 master_runner.cc:186] Error getting master registration for 127.0.115.190:37855: OK, instance_id { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" instance_seqno: 1790280631303043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:10:31.844604 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:10:31.848515 14035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:31.848706 14035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 20:10:31 dist-test-slave-2pm7 krb5kdc[13777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
I20260924 20:10:31.846099 13939 master_runner.cc:186] Error getting master registration for 127.0.115.190:37855: OK, instance_id { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" instance_seqno: 1790280631303043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:10:31.850011 13939 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:10:31.850203 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.830467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.0.115.189:40741)
0924 20:10:31.830736 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.830740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.830988 (+ 248us) server_negotiation.cc:407] Connection header received
0924 20:10:31.831180 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.831186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.831236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.831287 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.832187 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.832920 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.837371 (+ 4451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.837743 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.837842 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.846298 (+ 8456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.846303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.846305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.846331 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.848121 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.848613 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.848618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.848620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.848708 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.849049 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.849054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.849057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.849274 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.849411 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.849870 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.850015 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260924 20:10:31.850382 14034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:10:31.850466 14034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:10:31.853302 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.842810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:33335 (local address 127.0.0.1:56854)
0924 20:10:31.843044 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:31.843057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:31.843132 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:31.844187 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:31.844190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:31.844364 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:31.844477 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.844485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.845557 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.845561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.846496 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:31.846510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.847163 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.847166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:31.847245 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.848074 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:31.848090 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:31.849633 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:31.851675 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.851679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.851681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.851922 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.852271 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:31.852274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:31.852276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:31.852360 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:31.852807 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:31.852810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:31.852906 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 20:10:31.852991 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:10:31.853126 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
W20260924 20:10:31.853618 14046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:10:31.853721 14046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:10:31.853955 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:31.843259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.0.115.188:33335)
0924 20:10:31.843976 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:31.843981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:31.843992 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:10:31.844023 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:31.844025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:31.844063 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:31.844591 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:31.844687 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.845416 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.846640 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:31.847024 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:31.847157 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:31.849782 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:31.849788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:31.849794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:31.849822 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:31.851552 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.852069 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.852072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.852074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.852159 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:31.852491 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:31.852495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:31.852496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:31.852691 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:31.853624 (+ 933us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:31.853700 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 20:10:31.853812 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630}
I20260924 20:10:31.857333 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: Bootstrap starting.
I20260924 20:10:31.859704 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:31.860532 13858 log.cc:824] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:31.862361 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: No bootstrap required, opened a new log
I20260924 20:10:31.866398 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:31.866605 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:31.866668 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeadcb73ada14b548f21c9ea63fb7a3c, State: Initialized, Role: FOLLOWER
I20260924 20:10:31.867153 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:31.867652 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:31.867831 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:10:31.868467 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:10:31.875008 14061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:10:31.875141 14061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:10:31.878068 13790 master_runner.cc:263] No leader master found from master aeadcb73ada14b548f21c9ea63fb7a3c
I20260924 20:10:31.878129 13790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 20:10:32.021085 13961 tablet.cc:2369] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:32.043592 14023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:32.043807 14023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:32.045003 14023 leader_election.cc:288] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeadcb73ada14b548f21c9ea63fb7a3c (127.0.115.190:37855), bad5891e2739478387220cd56be68790 (127.0.115.188:33335)
W20260924 20:10:32.046257 13886 tablet.cc:2369] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:32.046234 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13b53b1789304b2881ad4a33bb9b88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" is_pre_election: true
I20260924 20:10:32.046581 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b53b1789304b2881ad4a33bb9b88d8 in term 0.
I20260924 20:10:32.047281 13882 leader_election.cc:302] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b53b1789304b2881ad4a33bb9b88d8, aeadcb73ada14b548f21c9ea63fb7a3c; no voters:
I20260924 20:10:32.047865 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13b53b1789304b2881ad4a33bb9b88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad5891e2739478387220cd56be68790" is_pre_election: true
I20260924 20:10:32.048249 13982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b53b1789304b2881ad4a33bb9b88d8 in term 0.
I20260924 20:10:32.048940 14023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:10:32.049021 14023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:10:32.049059 14023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 0 FOLLOWER]: Advancing to term 1
W20260924 20:10:32.049090 13811 tablet.cc:2369] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:32.050786 14023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:32.051132 14023 leader_election.cc:288] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [CANDIDATE]: Term 1 election: Requested vote from peers aeadcb73ada14b548f21c9ea63fb7a3c (127.0.115.190:37855), bad5891e2739478387220cd56be68790 (127.0.115.188:33335)
I20260924 20:10:32.051980 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13b53b1789304b2881ad4a33bb9b88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c"
I20260924 20:10:32.052112 13832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:32.053604 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b53b1789304b2881ad4a33bb9b88d8 in term 1.
I20260924 20:10:32.054011 13882 leader_election.cc:302] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b53b1789304b2881ad4a33bb9b88d8, aeadcb73ada14b548f21c9ea63fb7a3c; no voters:
I20260924 20:10:32.054265 13982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13b53b1789304b2881ad4a33bb9b88d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad5891e2739478387220cd56be68790"
I20260924 20:10:32.054395 13982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:32.055526 13982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b53b1789304b2881ad4a33bb9b88d8 in term 1.
I20260924 20:10:32.056033 14023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:32.056257 14023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [term 1 LEADER]: Becoming Leader. State: Replica: 13b53b1789304b2881ad4a33bb9b88d8, State: Running, Role: LEADER
I20260924 20:10:32.056625 14023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } }
I20260924 20:10:32.058538 14023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b53b1789304b2881ad4a33bb9b88d8. Latest consensus state: current_term: 1 leader_uuid: "13b53b1789304b2881ad4a33bb9b88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:32.058702 14023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:32.064257 14065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:32.065205 14065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:32.070063 13982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [term 1 FOLLOWER]: Refusing update from remote peer 13b53b1789304b2881ad4a33bb9b88d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:32.070770 14023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:32.070990 13832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [term 1 FOLLOWER]: Refusing update from remote peer 13b53b1789304b2881ad4a33bb9b88d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:32.071729 14063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:32.075412 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b53b1789304b2881ad4a33bb9b88d8. Latest consensus state: current_term: 1 leader_uuid: "13b53b1789304b2881ad4a33bb9b88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:32.075538 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:10:32.075623 14035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b53b1789304b2881ad4a33bb9b88d8. Latest consensus state: current_term: 1 leader_uuid: "13b53b1789304b2881ad4a33bb9b88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:32.075757 14035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:10:32.078756 14035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13b53b1789304b2881ad4a33bb9b88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:32.078884 14035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:10:32.078908 14067 mvcc.cc:205] Tried to move back new op lower bound from 7332989468952399872 to 7332989468907651072. Current Snapshot: MvccSnapshot[applied={T|T < 7332989468952399872}]
I20260924 20:10:32.078943 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13b53b1789304b2881ad4a33bb9b88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:32.079051 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:10:32.079356 14066 mvcc.cc:205] Tried to move back new op lower bound from 7332989468952399872 to 7332989468907651072. Current Snapshot: MvccSnapshot[applied={T|T < 7332989468952399872}]
I20260924 20:10:32.080111 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13b53b1789304b2881ad4a33bb9b88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:32.080251 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:32.080436 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13b53b1789304b2881ad4a33bb9b88d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeadcb73ada14b548f21c9ea63fb7a3c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37855 } } peers { permanent_uuid: "13b53b1789304b2881ad4a33bb9b88d8" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 40741 } } peers { permanent_uuid: "bad5891e2739478387220cd56be68790" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 33335 } } }
I20260924 20:10:32.080549 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:32.080826 14065 catalog_manager.cc:1347] Generated new cluster ID: a2c1da4eb2d34ee99ca8fec696672474
I20260924 20:10:32.080869 14065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:10:32.100523 14065 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:10:32.101935 14065 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:10:32.111629 14065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13b53b1789304b2881ad4a33bb9b88d8: Generated new TSK 0
I20260924 20:10:32.112586 14065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:10:32.855084 14046 catalog_manager.cc:1259] Loaded cluster ID: a2c1da4eb2d34ee99ca8fec696672474
I20260924 20:10:32.855213 14046 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: loading cluster ID for follower catalog manager: success
I20260924 20:10:32.856977 14046 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: acquiring CA information for follower catalog manager: success
I20260924 20:10:32.858177 14046 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bad5891e2739478387220cd56be68790: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:10:32.876487 14061 catalog_manager.cc:1259] Loaded cluster ID: a2c1da4eb2d34ee99ca8fec696672474
I20260924 20:10:32.876559 14061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: loading cluster ID for follower catalog manager: success
I20260924 20:10:32.878204 14061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: acquiring CA information for follower catalog manager: success
I20260924 20:10:32.879334 14061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aeadcb73ada14b548f21c9ea63fb7a3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:10:33.175051 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13790
I20260924 20:10:33.182320 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13865
I20260924 20:10:33.188747 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 13939
2026-09-24T20:10:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3241 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:10:33 dist-test-slave-2pm7 krb5kdc[14080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:10:33 dist-test-slave-2pm7 krb5kdc[14080](info): set up 2 sockets
Sep 24 20:10:33 dist-test-slave-2pm7 krb5kdc[14080](info): commencing operation
krb5kdc: starting...
W20260924 20:10:34.240375 462 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:10:34 dist-test-slave-2pm7 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:10:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:10:34.367412 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34415
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:34415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:34.479043 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:34.479311 14093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:34.479354 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:34.483434 14093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:34.483493 14093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:34.483513 14093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:34.483529 14093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:34.483546 14093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:34.488886 14093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:34415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:34.490453 14093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:34.491564 14093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:34.500696 14101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:34.500765 14093 server_base.cc:1034] running on GCE node
W20260924 20:10:34.500701 14099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:34.500711 14098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:34.501495 14093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:34.502604 14093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:34.503767 14093 hybrid_clock.cc:648] HybridClock initialized: now 1790280634503725 us; error 54 us; skew 500 ppm
Sep 24 20:10:34 dist-test-slave-2pm7 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:34.506486 14093 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:34.507356 14093 webserver.cc:460] Webserver started at http://127.0.115.190:36585/ using document root <none> and password file <none>
I20260924 20:10:34.507874 14093 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:34.507932 14093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:34.508184 14093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:34.510171 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "aa1f6f3921d345c5af1d0a6525119c26"
format_stamp: "Formatted at 2026-09-24 20:10:34 on dist-test-slave-2pm7"
server_key: "54d5d67c60beb591737502df72a0fa6b"
server_key_iv: "af9ba50fac820171ac552a1d95934b96"
server_key_version: "encryptionkey@0"
I20260924 20:10:34.510676 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "aa1f6f3921d345c5af1d0a6525119c26"
format_stamp: "Formatted at 2026-09-24 20:10:34 on dist-test-slave-2pm7"
server_key: "54d5d67c60beb591737502df72a0fa6b"
server_key_iv: "af9ba50fac820171ac552a1d95934b96"
server_key_version: "encryptionkey@0"
I20260924 20:10:34.514338 14093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 20:10:34.516875 14108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:34.518117 14093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:34.518261 14093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "aa1f6f3921d345c5af1d0a6525119c26"
format_stamp: "Formatted at 2026-09-24 20:10:34 on dist-test-slave-2pm7"
server_key: "54d5d67c60beb591737502df72a0fa6b"
server_key_iv: "af9ba50fac820171ac552a1d95934b96"
server_key_version: "encryptionkey@0"
I20260924 20:10:34.518374 14093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:34.580107 14093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:34.583137 14093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:34.583352 14093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:34.590374 14093 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:34415
I20260924 20:10:34.590406 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:34415 every 8 connection(s)
I20260924 20:10:34.591337 14093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:34.594209 14161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:34.594600 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14093
I20260924 20:10:34.594750 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:10:34.595065 462 external_mini_cluster.cc:1428] Setting key 7efffc564a949fbb595f28f5588ad041
I20260924 20:10:34.601598 14161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26: Bootstrap starting.
I20260924 20:10:34.604434 14161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26: Neither blocks nor log segments found. Creating new log.
Sep 24 20:10:34 dist-test-slave-2pm7 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:34.605320 14161 log.cc:824] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:34.607800 14161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26: No bootstrap required, opened a new log
I20260924 20:10:34.610412 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:34.596597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.0.115.190:34415)
0924 20:10:34.597039 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:34.597058 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:34.597101 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:10:34.597841 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:34.597876 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:34.598317 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:34.598686 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:34.599401 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:34.600991 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:34.601951 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:34.602605 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:34.602785 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:34.605357 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:34.605364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:34.605381 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:34.605416 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:34.607377 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:34.607804 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:34.607812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:34.607821 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:34.607990 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:34.608300 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:34.608303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:34.608305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:34.608853 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:34.609051 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:34.609338 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 20:10:34.609694 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":114,"threads_started":1}
I20260924 20:10:34.612093 14161 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1f6f3921d345c5af1d0a6525119c26" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34415 } }
I20260924 20:10:34.612401 14161 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:34.612504 14161 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1f6f3921d345c5af1d0a6525119c26, State: Initialized, Role: FOLLOWER
I20260924 20:10:34.613022 14161 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1f6f3921d345c5af1d0a6525119c26" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34415 } }
I20260924 20:10:34.613183 14161 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:10:34.613251 14161 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:10:34.613420 14161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:34.613473 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:10:34.614504 14161 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1f6f3921d345c5af1d0a6525119c26" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34415 } }
I20260924 20:10:34.614856 14161 leader_election.cc:302] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa1f6f3921d345c5af1d0a6525119c26; no voters:
I20260924 20:10:34.615236 14161 leader_election.cc:288] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:10:34.615339 14166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:34.615631 14166 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [term 1 LEADER]: Becoming Leader. State: Replica: aa1f6f3921d345c5af1d0a6525119c26, State: Running, Role: LEADER
I20260924 20:10:34.616025 14166 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1f6f3921d345c5af1d0a6525119c26" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34415 } }
I20260924 20:10:34.616540 14161 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:34.617805 14167 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa1f6f3921d345c5af1d0a6525119c26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1f6f3921d345c5af1d0a6525119c26" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34415 } } }
I20260924 20:10:34.617813 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa1f6f3921d345c5af1d0a6525119c26. Latest consensus state: current_term: 1 leader_uuid: "aa1f6f3921d345c5af1d0a6525119c26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1f6f3921d345c5af1d0a6525119c26" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34415 } } }
I20260924 20:10:34.618006 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:34.618377 14176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:34.618288 14167 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:34.621491 14176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:34.627615 14176 catalog_manager.cc:1347] Generated new cluster ID: 35dd48cb681742f3a61012c342c59e9f
I20260924 20:10:34.627691 14176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:10:34.643675 14176 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:10:34.644831 14176 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:10:34.650346 14176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa1f6f3921d345c5af1d0a6525119c26: Generated new TSK 0
I20260924 20:10:34.651163 14176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:10:34.669914 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14093
2026-09-24T20:10:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1479 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:10:34 dist-test-slave-2pm7 krb5kdc[14188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:10:34 dist-test-slave-2pm7 krb5kdc[14188](info): set up 2 sockets
Sep 24 20:10:34 dist-test-slave-2pm7 krb5kdc[14188](info): commencing operation
krb5kdc: starting...
W20260924 20:10:35.772444 462 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:10:35 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:10:35.899849 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39215
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:39215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:36.009745 14201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:36.010144 14201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:36.010218 14201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:36.014262 14201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:36.014361 14201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:36.014408 14201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:36.014468 14201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:36.014506 14201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:36.019696 14201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:39215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:36.020784 14201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:36.021808 14201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:10:36.030405 14201 server_base.cc:1034] running on GCE node
W20260924 20:10:36.030414 14206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.030411 14207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.030603 14209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:36.031222 14201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:36.032403 14201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:36.033622 14201 hybrid_clock.cc:648] HybridClock initialized: now 1790280636033604 us; error 29 us; skew 500 ppm
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:36.036405 14201 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:36.037333 14201 webserver.cc:460] Webserver started at http://127.0.115.190:45199/ using document root <none> and password file <none>
I20260924 20:10:36.037896 14201 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:36.037997 14201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:36.038231 14201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:36.040417 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "268cfa47ace14690a4f74581462e4c27"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "ba43dd26d6a3d3b0a83bdb767d5ef754"
server_key_iv: "6100d83dde379855260b5812d04b250e"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.041005 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "268cfa47ace14690a4f74581462e4c27"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "ba43dd26d6a3d3b0a83bdb767d5ef754"
server_key_iv: "6100d83dde379855260b5812d04b250e"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.044771 14201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:10:36.047171 14216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.048420 14201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:36.048565 14201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "268cfa47ace14690a4f74581462e4c27"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "ba43dd26d6a3d3b0a83bdb767d5ef754"
server_key_iv: "6100d83dde379855260b5812d04b250e"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.048679 14201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:36.067720 14201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:36.070783 14201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:36.071017 14201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:36.078516 14201 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:39215
I20260924 20:10:36.078819 14268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:39215 every 8 connection(s)
I20260924 20:10:36.079535 14201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:36.082415 14269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:36.085996 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14201
I20260924 20:10:36.086143 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:10:36.086416 462 external_mini_cluster.cc:1428] Setting key 9069f70cfc89f99a8211f15c5774dd7e
I20260924 20:10:36.088439 14269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Bootstrap starting.
I20260924 20:10:36.091220 14269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:36.092141 14269 log.cc:824] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:36.094385 14269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: No bootstrap required, opened a new log
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:36.098994 14269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:36.099291 14269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:36.099356 14269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268cfa47ace14690a4f74581462e4c27, State: Initialized, Role: FOLLOWER
I20260924 20:10:36.099864 14269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:36.100076 14269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:10:36.100148 14269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:10:36.100245 14269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:36.101429 14269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:36.101814 14269 leader_election.cc:302] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268cfa47ace14690a4f74581462e4c27; no voters:
I20260924 20:10:36.102149 14269 leader_election.cc:288] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:10:36.102322 14274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:36.102415 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.087856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.0.115.190:39215)
0924 20:10:36.088337 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.088357 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.088400 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:10:36.089223 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.089247 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.089720 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.090078 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:36.090799 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.092343 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.093407 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.094097 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.094263 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.097006 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:36.097013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:36.097038 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:36.097078 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:36.099039 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.099564 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.099572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.099579 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.099713 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.100012 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.100018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.100022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.100604 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:36.100787 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.101052 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.101376 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":120,"threads_started":1}
I20260924 20:10:36.102768 14274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 LEADER]: Becoming Leader. State: Replica: 268cfa47ace14690a4f74581462e4c27, State: Running, Role: LEADER
I20260924 20:10:36.103102 14274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:36.103431 14269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:36.105350 14275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "268cfa47ace14690a4f74581462e4c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } } }
I20260924 20:10:36.105475 14275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:36.106047 14276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268cfa47ace14690a4f74581462e4c27. Latest consensus state: current_term: 1 leader_uuid: "268cfa47ace14690a4f74581462e4c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } } }
I20260924 20:10:36.106143 14276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: This master's current role is: LEADER
W20260924 20:10:36.108188 14288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:10:36.108247 14288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:10:36.108328 14289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:36.109143 14289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:36.114156 14289 catalog_manager.cc:1347] Generated new cluster ID: 28b86c4a24af4ae29ba3b0f268af6479
I20260924 20:10:36.114217 14289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:10:36.126231 14289 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:10:36.127411 14289 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:10:36.137717 14289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Generated new TSK 0
I20260924 20:10:36.138542 14289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:10:36.171602 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:39215
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:36.283600 14295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:36.283910 14295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:36.284008 14295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:36.288062 14295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:36.288127 14295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:36.288210 14295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:10:36.293416 14295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:36.294437 14295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:36.295428 14295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:36.304926 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.304926 14303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.304943 14300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:36.305282 14295 server_base.cc:1034] running on GCE node
I20260924 20:10:36.305770 14295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:36.306409 14295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:36.307628 14295 hybrid_clock.cc:648] HybridClock initialized: now 1790280636307613 us; error 28 us; skew 500 ppm
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:36.310458 14295 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:10:36.311264 14295 webserver.cc:460] Webserver started at http://127.0.115.129:40375/ using document root <none> and password file <none>
I20260924 20:10:36.311789 14295 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:36.311844 14295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:36.312081 14295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:36.314129 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "d10e82e0c0404068afef23682fe029b5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "3032052a9c696b56060bb19a2692d034"
server_key_iv: "cfd4a2634749ce75f60a003882df9332"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.314663 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "d10e82e0c0404068afef23682fe029b5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "3032052a9c696b56060bb19a2692d034"
server_key_iv: "cfd4a2634749ce75f60a003882df9332"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.318348 14295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 20:10:36.320703 14310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.321914 14295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:10:36.322007 14295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "d10e82e0c0404068afef23682fe029b5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "3032052a9c696b56060bb19a2692d034"
server_key_iv: "cfd4a2634749ce75f60a003882df9332"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.322141 14295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:36.332363 14295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:36.335040 14295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:36.335237 14295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:36.335820 14295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:36.336668 14295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:36.336714 14295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.336786 14295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:36.336817 14295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.345932 14295 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:42537
I20260924 20:10:36.345983 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:42537 every 8 connection(s)
I20260924 20:10:36.346907 14295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:10:36.347769 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14295
I20260924 20:10:36.347918 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:10:36.348232 462 external_mini_cluster.cc:1428] Setting key 1a182f00b643417c2c219bb00cb8fa1e
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:36.366010 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.349878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:36531 (local address 127.0.115.190:39215)
0924 20:10:36.350003 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.350007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.351063 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 20:10:36.352702 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.352705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.352767 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.352836 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:36.354508 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.355041 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.356940 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.357158 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.357242 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.362264 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:36.362271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:36.362278 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:36.362316 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:36.363822 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.364425 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.364429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.364430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.364505 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.364862 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.364865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.364867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.365075 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:36.365183 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.365696 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.365829 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:10:36.366919 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.350352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:36531)
0924 20:10:36.350883 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:36.350943 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:36.352445 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:36.353032 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:36.353043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:36.353601 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:36.354338 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.354364 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.355178 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.355184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.356814 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.356828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.357327 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.357331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.357553 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.358190 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:36.358215 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:36.362075 (+ 3860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:36.363936 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:36.363992 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:36.364016 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:36.364302 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:36.364609 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:36.364612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:36.364614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:36.364756 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:36.365183 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:36.365192 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:36.365443 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 20:10:36.365639 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 20:10:36.365973 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260924 20:10:36.368310 14424 heartbeater.cc:347] Connected to a master server at 127.0.115.190:39215
I20260924 20:10:36.368578 14424 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:36.369098 14424 heartbeater.cc:511] Master 127.0.115.190:39215 requested a full tablet report, sending...
I20260924 20:10:36.370694 14233 ts_manager.cc:194] Registered new tserver with Master: d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537)
I20260924 20:10:36.371862 14233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:36531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:10:36.378965 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:39215
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:36.493731 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:36.494104 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:36.494158 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:36.498708 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:36.498772 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:36.498837 14429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:10:36.504814 14429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:36.505898 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:36.506911 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:36.516894 14435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.516894 14437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.516898 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:36.517127 14429 server_base.cc:1034] running on GCE node
I20260924 20:10:36.517617 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:36.518268 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:36.519465 14429 hybrid_clock.cc:648] HybridClock initialized: now 1790280636519442 us; error 59 us; skew 500 ppm
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:36.522246 14429 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:10:36.523118 14429 webserver.cc:460] Webserver started at http://127.0.115.130:35613/ using document root <none> and password file <none>
I20260924 20:10:36.523691 14429 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:36.523784 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:36.524083 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:36.526314 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "ff4c2143966e41f991d4983a7f44261e"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "c66d91524fb6481434bdf635beccf598"
server_key_iv: "d8d71afe7ed06c91602e54b95c5229cc"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.526885 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "ff4c2143966e41f991d4983a7f44261e"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "c66d91524fb6481434bdf635beccf598"
server_key_iv: "d8d71afe7ed06c91602e54b95c5229cc"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.530836 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:10:36.533439 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.534833 14429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:10:36.534976 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "ff4c2143966e41f991d4983a7f44261e"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "c66d91524fb6481434bdf635beccf598"
server_key_iv: "d8d71afe7ed06c91602e54b95c5229cc"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.535107 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:36.562913 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:36.566507 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:36.566717 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:36.567342 14429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:36.568271 14429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:36.568341 14429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.568413 14429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:36.568465 14429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.577837 14429 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:35883
I20260924 20:10:36.577867 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:35883 every 8 connection(s)
I20260924 20:10:36.578804 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:10:36.585660 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14429
I20260924 20:10:36.585772 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:10:36.586099 462 external_mini_cluster.cc:1428] Setting key ec47bb78659c623e1e97dc1f94e6dfb2
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:36.594014 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.580840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:40973 (local address 127.0.115.190:39215)
0924 20:10:36.580984 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.580988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.581900 (+ 912us) server_negotiation.cc:407] Connection header received
0924 20:10:36.583255 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.583260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.583320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.583435 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:36.584721 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.585258 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.586785 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.587056 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.587175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.589793 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:36.589797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:36.589799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:36.589823 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:36.591326 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.592055 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.592060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.592062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.592184 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.592641 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.592644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.592645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.592844 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:36.592925 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.593738 (+ 813us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.593866 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:10:36.595243 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.581257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:40973)
0924 20:10:36.581744 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:36.581786 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:36.583031 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:36.583626 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:36.583633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:36.584080 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:36.584567 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.584581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.585386 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.585390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.586437 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.586447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.587218 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.587222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.587438 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.588128 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:36.588146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:36.589616 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:36.591458 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:36.591467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:36.591488 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:36.591884 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:36.592304 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:36.592308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:36.592310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:36.592508 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:36.592965 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:36.592974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:36.593355 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 20:10:36.593631 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 20:10:36.594061 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":142,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:10:36.596640 14558 heartbeater.cc:347] Connected to a master server at 127.0.115.190:39215
I20260924 20:10:36.596903 14558 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:36.597450 14558 heartbeater.cc:511] Master 127.0.115.190:39215 requested a full tablet report, sending...
I20260924 20:10:36.598567 14233 ts_manager.cc:194] Registered new tserver with Master: ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:10:36.599221 14233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:40973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:10:36.615796 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:39215
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:36.730707 14563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:36.731038 14563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:36.731153 14563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:36.735162 14563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:36.735249 14563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:36.735344 14563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:10:36.740659 14563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:36.741729 14563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:36.742749 14563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:36.752321 14569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.752331 14568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:36.752331 14571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:36.752432 14563 server_base.cc:1034] running on GCE node
I20260924 20:10:36.753013 14563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:36.753732 14563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:36.754925 14563 hybrid_clock.cc:648] HybridClock initialized: now 1790280636754908 us; error 28 us; skew 500 ppm
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:36.764372 14563 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:10:36.765343 14563 webserver.cc:460] Webserver started at http://127.0.115.131:46181/ using document root <none> and password file <none>
I20260924 20:10:36.765887 14563 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:36.765944 14563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:36.766119 14563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:36.768226 14563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "7f667ab687704df5b37cbb7f99cc61d5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "8319b706cea9bdf8545f6175c85377c9"
server_key_iv: "bb93fc44aa733934ddc49cc78cc58a23"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.768745 14563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "7f667ab687704df5b37cbb7f99cc61d5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "8319b706cea9bdf8545f6175c85377c9"
server_key_iv: "bb93fc44aa733934ddc49cc78cc58a23"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.772411 14563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:10:36.774744 14578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.776044 14563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:36.776175 14563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "7f667ab687704df5b37cbb7f99cc61d5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "8319b706cea9bdf8545f6175c85377c9"
server_key_iv: "bb93fc44aa733934ddc49cc78cc58a23"
server_key_version: "encryptionkey@0"
I20260924 20:10:36.776278 14563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:36.787727 14563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:36.790938 14563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:36.791157 14563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:36.791836 14563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:36.792713 14563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:36.792759 14563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.792826 14563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:36.792860 14563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:36.802663 14563 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44851
I20260924 20:10:36.802682 14691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44851 every 8 connection(s)
I20260924 20:10:36.803639 14563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:10:36.812364 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14563
I20260924 20:10:36.812536 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:10:36.812824 462 external_mini_cluster.cc:1428] Setting key a9339d2ce48397d27e754b5fe2795de3
Sep 24 20:10:36 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:36.818177 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.805654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:36203 (local address 127.0.115.190:39215)
0924 20:10:36.805794 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.805798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.806667 (+ 869us) server_negotiation.cc:407] Connection header received
0924 20:10:36.808249 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.808253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.808314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.808410 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:36.809584 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.810120 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.811233 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.811453 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.811567 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.814969 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:36.814974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:36.814977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:36.815007 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:36.816358 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.816837 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.816840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.816841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.816919 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.817227 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.817229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.817230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.817406 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:36.817482 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.817950 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.818058 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:10:36.818988 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.806075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:36203)
0924 20:10:36.806515 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:36.806560 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:36.807923 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:36.808566 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:36.808573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:36.808922 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:36.809422 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.809435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.810228 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.810231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.811110 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.811118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.811617 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.811620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.811757 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.812434 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:36.812451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:36.814266 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:36.816460 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:36.816465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:36.816480 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:36.816739 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:36.817002 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:36.817004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:36.817006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:36.817145 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:36.817492 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:36.817497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:36.817729 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 20:10:36.817886 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 20:10:36.818152 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
I20260924 20:10:36.820119 14692 heartbeater.cc:347] Connected to a master server at 127.0.115.190:39215
I20260924 20:10:36.820384 14692 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:36.820852 14692 heartbeater.cc:511] Master 127.0.115.190:39215 requested a full tablet report, sending...
I20260924 20:10:36.821853 14233 ts_manager.cc:194] Registered new tserver with Master: 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851)
I20260924 20:10:36.822371 14233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:36203
I20260924 20:10:36.827078 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:10:36.836567 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.829114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.0.115.190:39215)
0924 20:10:36.829238 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.829245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.829402 (+ 157us) server_negotiation.cc:407] Connection header received
0924 20:10:36.829524 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.829526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.829584 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.829650 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:36.830373 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.830949 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.831771 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.832100 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.832201 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.833550 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:36.833553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:36.833555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:36.833577 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:36.834918 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.835299 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.835302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.835303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.835380 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:36.835679 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:36.835681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:36.835682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:36.835902 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:36.836107 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.836275 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.836373 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:10:36.839749 14233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:10:36.843078 14233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:10:36.859364 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.852237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:42537 (local address 127.0.0.1:51336)
0924 20:10:36.853182 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:36.853237 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:36.853395 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:36.854308 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:36.854311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:36.854328 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:36.854498 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.854508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.856075 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.856078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.857594 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.857604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.858727 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.858731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.858847 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.858894 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 20:10:36.859028 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 20:10:36.859164 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":139,"spinlock_wait_cycles":4480,"thread_start_us":62,"threads_started":1}
I20260924 20:10:36.859648 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.853280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.0.115.130:35883)
0924 20:10:36.853712 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.853723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.853746 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:10:36.853827 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.853834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.854039 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.854216 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:36.854734 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.855717 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.857855 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.858627 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.858720 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.858865 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.859385 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.859491 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":210,"threads_started":1}
I20260924 20:10:36.859992 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.852344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.115.129:42537)
0924 20:10:36.852908 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.853845 (+ 937us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.853869 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:10:36.853936 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.853940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.854105 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.854656 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:36.854827 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.855894 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.857723 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.858592 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.858714 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.858865 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.859081 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.859189 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":55,"threads_started":1}
I20260924 20:10:36.860000 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.852633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35883 (local address 127.0.0.1:44592)
0924 20:10:36.853632 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:36.853645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:36.853733 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:36.854393 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:36.854396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:36.854424 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:36.854603 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.854609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.855859 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.855866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.857721 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.857733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.858938 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.858941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.859039 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.859064 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:10:36.859133 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:10:36.859819 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":209,"thread_start_us":95,"threads_started":1}
I20260924 20:10:36.860322 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.852339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44851 (local address 127.0.0.1:37616)
0924 20:10:36.853182 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:36.853239 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:36.853396 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:36.854203 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:36.854209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:36.854234 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:36.854395 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.854402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.855934 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.855961 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.858222 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:36.858230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.859527 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.859533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:36.859624 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.859645 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:10:36.859718 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 20:10:36.859797 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":147,"spinlock_wait_cycles":11520,"thread_start_us":133,"threads_started":1}
I20260924 20:10:36.860689 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:36.853064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.0.115.131:44851)
0924 20:10:36.853604 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:36.853616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:36.853644 (+ 28us) server_negotiation.cc:407] Connection header received
0924 20:10:36.853732 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:36.853738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:36.854035 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:36.854212 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:36.854539 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.855779 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.858356 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:36.859197 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:36.859303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:36.859448 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:36.859910 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 20:10:36.860063 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":177,"threads_started":1}
I20260924 20:10:36.862924 14492 tablet_service.cc:1467] Processing CreateTablet for tablet 625f776b9e964b1397ab821cc42d638c (DEFAULT_TABLE table=test-table [id=fe25aa27bcf84141a47fbe9c10130784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:36.862994 14358 tablet_service.cc:1467] Processing CreateTablet for tablet 625f776b9e964b1397ab821cc42d638c (DEFAULT_TABLE table=test-table [id=fe25aa27bcf84141a47fbe9c10130784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:36.863276 14626 tablet_service.cc:1467] Processing CreateTablet for tablet 625f776b9e964b1397ab821cc42d638c (DEFAULT_TABLE table=test-table [id=fe25aa27bcf84141a47fbe9c10130784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:36.864121 14358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625f776b9e964b1397ab821cc42d638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:36.864125 14492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625f776b9e964b1397ab821cc42d638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:36.864168 14626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625f776b9e964b1397ab821cc42d638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:36.870615 14709 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Bootstrap starting.
I20260924 20:10:36.872020 14711 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Bootstrap starting.
I20260924 20:10:36.872048 14710 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Bootstrap starting.
I20260924 20:10:36.873637 14709 tablet_bootstrap.cc:654] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:36.874100 14711 tablet_bootstrap.cc:654] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:36.874259 14710 tablet_bootstrap.cc:654] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:36.874668 14709 log.cc:824] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:36.874946 14711 log.cc:824] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:36.875144 14710 log.cc:824] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:36.876813 14709 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: No bootstrap required, opened a new log
I20260924 20:10:36.876813 14711 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: No bootstrap required, opened a new log
I20260924 20:10:36.876971 14710 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: No bootstrap required, opened a new log
I20260924 20:10:36.877005 14709 ts_tablet_manager.cc:1397] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:10:36.877005 14711 ts_tablet_manager.cc:1397] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:10:36.877197 14710 ts_tablet_manager.cc:1397] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:10:36.880921 14709 raft_consensus.cc:348] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:36.881007 14710 raft_consensus.cc:348] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:36.881301 14710 raft_consensus.cc:374] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:36.881321 14709 raft_consensus.cc:374] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:36.881374 14709 raft_consensus.cc:729] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4c2143966e41f991d4983a7f44261e, State: Initialized, Role: FOLLOWER
I20260924 20:10:36.881476 14710 raft_consensus.cc:729] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e82e0c0404068afef23682fe029b5, State: Initialized, Role: FOLLOWER
I20260924 20:10:36.881247 14711 raft_consensus.cc:348] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:36.881626 14711 raft_consensus.cc:374] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:36.881695 14711 raft_consensus.cc:729] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f667ab687704df5b37cbb7f99cc61d5, State: Initialized, Role: FOLLOWER
I20260924 20:10:36.881927 14709 consensus_queue.cc:260] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:36.882059 14711 consensus_queue.cc:260] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:36.882073 14710 consensus_queue.cc:260] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:36.882972 14710 ts_tablet_manager.cc:1428] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:10:36.882997 14709 ts_tablet_manager.cc:1428] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:10:36.882997 14711 ts_tablet_manager.cc:1428] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:10:36.883373 14424 heartbeater.cc:503] Master 127.0.115.190:39215 was elected leader, sending a full tablet report...
I20260924 20:10:36.883594 14692 heartbeater.cc:503] Master 127.0.115.190:39215 was elected leader, sending a full tablet report...
I20260924 20:10:36.884105 14558 heartbeater.cc:503] Master 127.0.115.190:39215 was elected leader, sending a full tablet report...
W20260924 20:10:37.054715 14693 tablet.cc:2369] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:10:37.080084 14559 tablet.cc:2369] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:10:37.098552 14425 tablet.cc:2369] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:37.236415 14716 raft_consensus.cc:482] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:37.236670 14716 raft_consensus.cc:504] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:37.237850 14716 leader_election.cc:288] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537), ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883)
I20260924 20:10:37.242020 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.238347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:48457 (local address 127.0.115.130:35883)
0924 20:10:37.238468 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:37.238471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:37.238484 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:10:37.238560 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:37.238562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:37.238635 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:37.238731 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:37.239091 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.239787 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.240997 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.241426 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.241554 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.241613 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:37.241806 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 20:10:37.241883 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:10:37.242043 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.238237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:42603 (local address 127.0.115.129:42537)
0924 20:10:37.238397 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:37.238402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:37.238912 (+ 510us) server_negotiation.cc:407] Connection header received
0924 20:10:37.238997 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:37.239003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:37.239069 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:37.239143 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:37.239410 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.240045 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.241091 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.241454 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.241590 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.241645 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:37.241858 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 20:10:37.241938 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 20:10:37.242040 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.238121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35883 (local address 127.0.115.131:48457)
0924 20:10:37.238229 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:37.238245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:37.238369 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:37.238764 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:37.238767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:37.238820 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:37.238968 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.238976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.239889 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.239893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.240859 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.240869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.241546 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.241552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.241665 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.241691 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:10:37.241758 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:10:37.241846 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":57}
I20260924 20:10:37.242058 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.238121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:42537 (local address 127.0.115.131:42603)
0924 20:10:37.238805 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:37.238821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:37.238918 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:37.239170 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:37.239173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:37.239189 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:37.239301 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.239308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.240147 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.240149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.240948 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.240961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.241606 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.241609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.241711 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.241728 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:10:37.241825 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 20:10:37.241889 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":329,"thread_start_us":92,"threads_started":1}
I20260924 20:10:37.242692 14378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625f776b9e964b1397ab821cc42d638c" candidate_uuid: "7f667ab687704df5b37cbb7f99cc61d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d10e82e0c0404068afef23682fe029b5" is_pre_election: true
I20260924 20:10:37.242743 14512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625f776b9e964b1397ab821cc42d638c" candidate_uuid: "7f667ab687704df5b37cbb7f99cc61d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4c2143966e41f991d4983a7f44261e" is_pre_election: true
I20260924 20:10:37.243016 14378 raft_consensus.cc:2454] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f667ab687704df5b37cbb7f99cc61d5 in term 0.
I20260924 20:10:37.243039 14512 raft_consensus.cc:2454] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f667ab687704df5b37cbb7f99cc61d5 in term 0.
I20260924 20:10:37.243455 14579 leader_election.cc:302] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f667ab687704df5b37cbb7f99cc61d5, d10e82e0c0404068afef23682fe029b5; no voters:
I20260924 20:10:37.243738 14716 raft_consensus.cc:2781] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:10:37.243824 14716 raft_consensus.cc:482] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:10:37.243866 14716 raft_consensus.cc:3037] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:37.244870 14716 raft_consensus.cc:504] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:37.245229 14716 leader_election.cc:288] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [CANDIDATE]: Term 1 election: Requested vote from peers d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537), ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883)
I20260924 20:10:37.245669 14512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625f776b9e964b1397ab821cc42d638c" candidate_uuid: "7f667ab687704df5b37cbb7f99cc61d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4c2143966e41f991d4983a7f44261e"
I20260924 20:10:37.245709 14378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625f776b9e964b1397ab821cc42d638c" candidate_uuid: "7f667ab687704df5b37cbb7f99cc61d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d10e82e0c0404068afef23682fe029b5"
I20260924 20:10:37.245837 14378 raft_consensus.cc:3037] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:37.245790 14512 raft_consensus.cc:3037] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:37.246841 14378 raft_consensus.cc:2454] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f667ab687704df5b37cbb7f99cc61d5 in term 1.
I20260924 20:10:37.247004 14512 raft_consensus.cc:2454] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f667ab687704df5b37cbb7f99cc61d5 in term 1.
I20260924 20:10:37.247133 14579 leader_election.cc:302] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f667ab687704df5b37cbb7f99cc61d5, d10e82e0c0404068afef23682fe029b5; no voters:
I20260924 20:10:37.247326 14716 raft_consensus.cc:2781] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:37.247548 14716 raft_consensus.cc:686] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 LEADER]: Becoming Leader. State: Replica: 7f667ab687704df5b37cbb7f99cc61d5, State: Running, Role: LEADER
I20260924 20:10:37.247928 14716 consensus_queue.cc:237] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:37.251366 14232 catalog_manager.cc:5495] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131). New cstate: current_term: 1 leader_uuid: "7f667ab687704df5b37cbb7f99cc61d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } health_report { overall_health: UNKNOWN } } }
I20260924 20:10:37.309492 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.305998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.0.115.131:44851)
0924 20:10:37.306128 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:37.306132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:37.306146 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:10:37.306262 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:37.306264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:37.306320 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:37.306413 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:10:37.306705 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.307227 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.308074 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.308287 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.308366 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.308673 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:10:37.309114 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:10:37.309206 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:37.309289 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 20:10:37.309343 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:10:37.315543 14378 raft_consensus.cc:1261] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Refusing update from remote peer 7f667ab687704df5b37cbb7f99cc61d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:37.315562 14512 raft_consensus.cc:1261] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 1 FOLLOWER]: Refusing update from remote peer 7f667ab687704df5b37cbb7f99cc61d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:37.316285 14716 consensus_queue.cc:1035] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:37.316501 14719 consensus_queue.cc:1035] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:37.326546 14723 mvcc.cc:205] Tried to move back new op lower bound from 7332989490436730880 to 7332989490170441728. Current Snapshot: MvccSnapshot[applied={T|T < 7332989490436730880 or (T in {7332989490436730880})}]
I20260924 20:10:37.331501 14232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:37.331678 14232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:37.334149 14232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Sending DeleteTablet for 3 replicas of tablet 625f776b9e964b1397ab821cc42d638c
I20260924 20:10:37.334965 14626 tablet_service.cc:1514] Processing DeleteTablet for tablet 625f776b9e964b1397ab821cc42d638c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:37 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:37616
I20260924 20:10:37.335089 14358 tablet_service.cc:1514] Processing DeleteTablet for tablet 625f776b9e964b1397ab821cc42d638c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:37 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:51336
I20260924 20:10:37.335093 14492 tablet_service.cc:1514] Processing DeleteTablet for tablet 625f776b9e964b1397ab821cc42d638c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:37 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:44592
I20260924 20:10:37.335598 14733 tablet_replica.cc:330] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: stopping tablet replica
I20260924 20:10:37.335729 14732 tablet_replica.cc:330] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: stopping tablet replica
I20260924 20:10:37.335758 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14295
I20260924 20:10:37.336848 14734 tablet_replica.cc:330] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: stopping tablet replica
I20260924 20:10:37.337205 14734 raft_consensus.cc:2229] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:10:37.337543 14734 raft_consensus.cc:2258] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:37.339083 14734 ts_tablet_manager.cc:1905] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:37.342456 14734 ts_tablet_manager.cc:1918] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:37.342573 14734 log.cc:1196] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/625f776b9e964b1397ab821cc42d638c
I20260924 20:10:37.342928 14734 ts_tablet_manager.cc:1939] T 625f776b9e964b1397ab821cc42d638c P ff4c2143966e41f991d4983a7f44261e: Deleting consensus metadata
I20260924 20:10:37.344254 14732 raft_consensus.cc:2229] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:10:37.344419 14220 catalog_manager.cc:4841] TS ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883): tablet 625f776b9e964b1397ab821cc42d638c (table test-table [id=fe25aa27bcf84141a47fbe9c10130784]) successfully deleted
I20260924 20:10:37.344636 14732 raft_consensus.cc:2258] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:10:37.345310 14217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:42537 (error 108)
I20260924 20:10:37.346158 14732 ts_tablet_manager.cc:1905] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:37.346302 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.345980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:42537 (local address 127.0.0.1:51346)
0924 20:10:37.346120 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:37.346198 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 20:10:37.346407 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14429
I20260924 20:10:37.348804 14732 ts_tablet_manager.cc:1918] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:37.348904 14732 log.cc:1196] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/625f776b9e964b1397ab821cc42d638c
I20260924 20:10:37.349260 14732 ts_tablet_manager.cc:1939] T 625f776b9e964b1397ab821cc42d638c P 7f667ab687704df5b37cbb7f99cc61d5: Deleting consensus metadata
I20260924 20:10:37.350643 14218 catalog_manager.cc:4841] TS 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851): tablet 625f776b9e964b1397ab821cc42d638c (table test-table [id=fe25aa27bcf84141a47fbe9c10130784]) successfully deleted
W20260924 20:10:37.353564 14217 catalog_manager.cc:4584] TS d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 625f776b9e964b1397ab821cc42d638c: Network error: Client connection negotiation failed: client connection to 127.0.115.129:42537: connect: Connection refused (error 111)
I20260924 20:10:37.354090 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14563
I20260924 20:10:37.359369 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14201
I20260924 20:10:37.365018 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39215
--webserver_interface=127.0.115.190
--webserver_port=45199
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:39215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:37.477341 14736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:37.477623 14736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:37.477664 14736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:37.481784 14736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:37.481957 14736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:37.482005 14736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:37.482033 14736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:37.482060 14736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:37.489894 14736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:39215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:37.491088 14736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:37.492224 14736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:37.507725 14743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:37.507756 14736 server_base.cc:1034] running on GCE node
W20260924 20:10:37.507731 14742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:37.507766 14745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:37.508786 14736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:37.509990 14736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:37.511183 14736 hybrid_clock.cc:648] HybridClock initialized: now 1790280637511164 us; error 29 us; skew 500 ppm
Sep 24 20:10:37 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:37.514070 14736 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:37.514971 14736 webserver.cc:460] Webserver started at http://127.0.115.190:45199/ using document root <none> and password file <none>
I20260924 20:10:37.515604 14736 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:37.515722 14736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:37.519632 14736 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:37.521988 14752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:37.523332 14736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 20:10:37.523478 14736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "268cfa47ace14690a4f74581462e4c27"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "ba43dd26d6a3d3b0a83bdb767d5ef754"
server_key_iv: "6100d83dde379855260b5812d04b250e"
server_key_version: "encryptionkey@0"
I20260924 20:10:37.524168 14736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:37.541265 14736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:37.544140 14736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:37.544348 14736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:37.551296 14736 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:39215
I20260924 20:10:37.551617 14804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:39215 every 8 connection(s)
I20260924 20:10:37.552455 14736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:37.556995 14805 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:10:37.558605 14805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Bootstrap starting.
I20260924 20:10:37.561748 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14736
I20260924 20:10:37.562181 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:42537
--local_ip_for_outbound_sockets=127.0.115.129
--tserver_master_addrs=127.0.115.190:39215
--webserver_port=40375
--webserver_interface=127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:10:37.576768 14805 log.cc:824] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:37.585628 14805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:10:37.586014 14805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Bootstrap complete.
I20260924 20:10:37.590269 14805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:37.590764 14805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268cfa47ace14690a4f74581462e4c27, State: Initialized, Role: FOLLOWER
I20260924 20:10:37.591248 14805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:37.591396 14805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:10:37.591492 14805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:10:37.591616 14805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:10:37.593571 14805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:37.593905 14805 leader_election.cc:302] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268cfa47ace14690a4f74581462e4c27; no voters:
I20260924 20:10:37.594247 14805 leader_election.cc:288] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:10:37.594331 14809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:10:37.594555 14809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 LEADER]: Becoming Leader. State: Replica: 268cfa47ace14690a4f74581462e4c27, State: Running, Role: LEADER
I20260924 20:10:37.594841 14809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:37.595504 14805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:37.595799 14809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268cfa47ace14690a4f74581462e4c27. Latest consensus state: current_term: 2 leader_uuid: "268cfa47ace14690a4f74581462e4c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } } }
I20260924 20:10:37.595918 14809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:37.596333 14810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "268cfa47ace14690a4f74581462e4c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } } }
I20260924 20:10:37.596536 14810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:37.597965 14813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:37.603135 14813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:37.603760 14813 catalog_manager.cc:1259] Loaded cluster ID: 28b86c4a24af4ae29ba3b0f268af6479
I20260924 20:10:37.603826 14813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:10:37.605185 14813 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 20:10:37.609833 14813 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 20:10:37.615626 14813 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:10:37.616441 14813 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Loaded TSK: 0
I20260924 20:10:37.617647 14813 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:10:37.681975 14807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:37.682273 14807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:37.682348 14807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:37.686421 14807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:37.686504 14807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:37.686592 14807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:10:37.691865 14807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:42537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=40375
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:37.692941 14807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:37.693977 14807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:37.704847 14832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:37.704885 14835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:37.705199 14833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:37.705358 14807 server_base.cc:1034] running on GCE node
I20260924 20:10:37.705881 14807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:37.706694 14807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:37.707911 14807 hybrid_clock.cc:648] HybridClock initialized: now 1790280637707864 us; error 55 us; skew 500 ppm
Sep 24 20:10:37 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:37.711318 14807 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:10:37.712496 14807 webserver.cc:460] Webserver started at http://127.0.115.129:40375/ using document root <none> and password file <none>
I20260924 20:10:37.713233 14807 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:37.713342 14807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:37.718453 14807 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:10:37.721120 14842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:37.722615 14807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:10:37.722733 14807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "d10e82e0c0404068afef23682fe029b5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "3032052a9c696b56060bb19a2692d034"
server_key_iv: "cfd4a2634749ce75f60a003882df9332"
server_key_version: "encryptionkey@0"
I20260924 20:10:37.723464 14807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:37.748956 14807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:37.752543 14807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:37.752738 14807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:37.753528 14807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:37.754845 14849 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:10:37.759119 14807 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:10:37.759181 14807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 20:10:37.759234 14807 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:10:37.760933 14807 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:10:37.760978 14807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 20:10:37.761314 14849 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Bootstrap starting.
I20260924 20:10:37.771446 14807 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:42537
I20260924 20:10:37.771560 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:42537 every 8 connection(s)
I20260924 20:10:37.772492 14807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:10:37.779590 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14807
I20260924 20:10:37.780133 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:35883
--local_ip_for_outbound_sockets=127.0.115.130
--tserver_master_addrs=127.0.115.190:39215
--webserver_port=35613
--webserver_interface=127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:10:37.780220 14849 log.cc:824] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:10:37 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:37.788936 14849 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:10:37.789381 14849 tablet_bootstrap.cc:492] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Bootstrap complete.
I20260924 20:10:37.790764 14849 ts_tablet_manager.cc:1397] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260924 20:10:37.791316 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.774696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:48531)
0924 20:10:37.775202 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:37.775244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:37.776132 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:37.777243 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:37.777251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:37.777674 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:37.778135 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.778149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.779454 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.779459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.780665 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.780678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.781820 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.781824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.782029 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.782681 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:37.782699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:37.784504 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:37.787484 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:37.787490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:37.787504 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:37.787853 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:37.788337 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:37.788340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:37.788342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:37.788509 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:37.789344 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:37.789354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:37.789696 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 20:10:37.789965 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 20:10:37.790341 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":134,"threads_started":1}
I20260924 20:10:37.792158 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.774645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:48531 (local address 127.0.115.190:39215)
0924 20:10:37.775015 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:37.775036 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:37.775368 (+ 332us) server_negotiation.cc:407] Connection header received
0924 20:10:37.776560 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:37.776570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:37.776861 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:37.777190 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:37.778322 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.779301 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.780816 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.781645 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.781828 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.784812 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:37.784823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:37.784844 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:37.784887 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:37.787336 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:37.788024 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:37.788033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:37.788043 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:37.788219 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:37.788646 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:37.788652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:37.788657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:37.789206 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:37.789364 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:37.790150 (+ 786us) server_negotiation.cc:299] Negotiation successful
0924 20:10:37.790527 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":125,"threads_started":1}
I20260924 20:10:37.793519 14849 raft_consensus.cc:348] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:37.794044 14849 raft_consensus.cc:729] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e82e0c0404068afef23682fe029b5, State: Initialized, Role: FOLLOWER
I20260924 20:10:37.794564 14849 consensus_queue.cc:260] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } }
I20260924 20:10:37.794669 14957 heartbeater.cc:347] Connected to a master server at 127.0.115.190:39215
I20260924 20:10:37.794941 14957 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:37.795472 14957 heartbeater.cc:511] Master 127.0.115.190:39215 requested a full tablet report, sending...
I20260924 20:10:37.795673 14849 ts_tablet_manager.cc:1428] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:10:37.798318 14769 ts_manager.cc:194] Registered new tserver with Master: d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537)
I20260924 20:10:37.802449 14769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:48531
I20260924 20:10:37.804975 14957 heartbeater.cc:503] Master 127.0.115.190:39215 was elected leader, sending a full tablet report...
Sep 24 20:10:37 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:10:37.812945 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.801807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.115.129:42537)
0924 20:10:37.802190 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:37.802202 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:37.802579 (+ 377us) server_negotiation.cc:407] Connection header received
0924 20:10:37.802811 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:37.802816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:37.802974 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:37.803120 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:37.803854 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.805164 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.806325 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.806672 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.806792 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.809295 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:37.809303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:37.809313 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:37.809337 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:37.810921 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:37.811426 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:37.811432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:37.811433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:37.811509 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:37.811813 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:37.811818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:37.811819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:37.812141 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:37.812277 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:37.812638 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 20:10:37.812802 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":288,"thread_start_us":99,"threads_started":1}
I20260924 20:10:37.812937 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.801755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:42537 (local address 127.0.0.1:51352)
0924 20:10:37.802435 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:37.802468 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:37.802653 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:37.803122 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:37.803128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:37.803576 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:37.803719 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.803728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.805286 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.805290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.806201 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.806211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.806793 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.806797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.806912 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.807514 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:37.807526 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:37.809143 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:37.811049 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:37.811058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:37.811063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:37.811300 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:37.811624 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:37.811628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:37.811630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:37.811721 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:37.812264 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:37.812270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:37.812399 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 20:10:37.812537 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 20:10:37.812746 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":125,"threads_started":1}
I20260924 20:10:37.813982 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet 625f776b9e964b1397ab821cc42d638c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:37 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:51352
I20260924 20:10:37.814838 14966 tablet_replica.cc:330] stopping tablet replica
I20260924 20:10:37.815150 14966 raft_consensus.cc:2229] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:10:37.815405 14966 raft_consensus.cc:2258] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:37.816439 14966 ts_tablet_manager.cc:1905] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:37.819856 14966 ts_tablet_manager.cc:1918] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:37.819984 14966 log.cc:1196] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/625f776b9e964b1397ab821cc42d638c
I20260924 20:10:37.820271 14966 ts_tablet_manager.cc:1939] T 625f776b9e964b1397ab821cc42d638c P d10e82e0c0404068afef23682fe029b5: Deleting consensus metadata
I20260924 20:10:37.821946 14753 catalog_manager.cc:4841] TS d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537): tablet 625f776b9e964b1397ab821cc42d638c (table test-table [id=fe25aa27bcf84141a47fbe9c10130784]) successfully deleted
W20260924 20:10:37.902371 14962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:37.902675 14962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:37.902767 14962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:37.907021 14962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:37.907104 14962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:37.907191 14962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:10:37.912710 14962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=35613
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:37.913836 14962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:37.914896 14962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:37.924927 14971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:37.924939 14972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:37.924939 14974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:37.925369 14962 server_base.cc:1034] running on GCE node
I20260924 20:10:37.925784 14962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:37.926496 14962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:37.927640 14962 hybrid_clock.cc:648] HybridClock initialized: now 1790280637927610 us; error 41 us; skew 500 ppm
Sep 24 20:10:37 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:37.930508 14962 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:10:37.931478 14962 webserver.cc:460] Webserver started at http://127.0.115.130:35613/ using document root <none> and password file <none>
I20260924 20:10:37.932137 14962 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:37.932240 14962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:37.936168 14962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:10:37.938275 14981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:37.939764 14962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:10:37.939982 14962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "ff4c2143966e41f991d4983a7f44261e"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "c66d91524fb6481434bdf635beccf598"
server_key_iv: "d8d71afe7ed06c91602e54b95c5229cc"
server_key_version: "encryptionkey@0"
I20260924 20:10:37.940625 14962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:37.966943 14962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:37.970494 14962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:37.970741 14962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:37.971441 14962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:37.972406 14962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:37.972496 14962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:37.972581 14962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:37.972630 14962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:37.982229 14962 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:35883
I20260924 20:10:37.982291 15094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:35883 every 8 connection(s)
I20260924 20:10:37.983363 14962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:10:37.987883 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 14962
I20260924 20:10:37.988301 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:44851
--local_ip_for_outbound_sockets=127.0.115.131
--tserver_master_addrs=127.0.115.190:39215
--webserver_port=46181
--webserver_interface=127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:10:37 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:37.998793 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.985306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:32971 (local address 127.0.115.190:39215)
0924 20:10:37.985495 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:37.985501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:37.986391 (+ 890us) server_negotiation.cc:407] Connection header received
0924 20:10:37.987660 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:37.987662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:37.987714 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:37.987843 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:37.989201 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.990014 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.991251 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.991700 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.991826 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.994594 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:37.994601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:37.994607 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:37.994647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:37.996436 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:37.996940 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:37.996946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:37.996951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:37.997066 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:37.997451 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:37.997458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:37.997462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:37.997743 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:37.997865 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:37.998370 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 20:10:37.998547 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:10:37.999483 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:37.985766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:32971)
0924 20:10:37.986231 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:37.986273 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:37.987435 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:37.988094 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:37.988102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:37.988545 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:37.989043 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.989058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.990165 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.990168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.991129 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:37.991140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:37.991850 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:37.991853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:37.992059 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:37.992665 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:37.992681 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:37.994399 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:37.996564 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:37.996569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:37.996581 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:37.996836 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:37.997169 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:37.997171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:37.997172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:37.997351 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:37.997862 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:37.997867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:37.998116 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 20:10:37.998276 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 20:10:37.998527 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260924 20:10:38.000759 15095 heartbeater.cc:347] Connected to a master server at 127.0.115.190:39215
I20260924 20:10:38.001014 15095 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:38.001516 15095 heartbeater.cc:511] Master 127.0.115.190:39215 requested a full tablet report, sending...
I20260924 20:10:38.002619 14769 ts_manager.cc:194] Registered new tserver with Master: ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883)
I20260924 20:10:38.003353 14769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:32971
W20260924 20:10:38.101504 15098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:38.101755 15098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:38.101797 15098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:38.105751 15098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:38.105805 15098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:38.105863 15098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:10:38.110944 15098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:44851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=46181
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:38.111907 15098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:38.112996 15098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:38.122795 15103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:38.122874 15098 server_base.cc:1034] running on GCE node
W20260924 20:10:38.122783 15104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:38.122800 15106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:38.123409 15098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:38.124105 15098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:38.125288 15098 hybrid_clock.cc:648] HybridClock initialized: now 1790280638125251 us; error 48 us; skew 500 ppm
Sep 24 20:10:38 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:38.128067 15098 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:10:38.129061 15098 webserver.cc:460] Webserver started at http://127.0.115.131:46181/ using document root <none> and password file <none>
I20260924 20:10:38.129622 15098 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:38.129695 15098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:38.133610 15098 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:10:38.135658 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:38.136950 15098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:10:38.137096 15098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "7f667ab687704df5b37cbb7f99cc61d5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "8319b706cea9bdf8545f6175c85377c9"
server_key_iv: "bb93fc44aa733934ddc49cc78cc58a23"
server_key_version: "encryptionkey@0"
I20260924 20:10:38.137650 15098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:38.167465 15098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:38.170416 15098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:38.170632 15098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:38.171267 15098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:38.172219 15098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:38.172288 15098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:38.172363 15098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:38.172416 15098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:38.181854 15098 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44851
I20260924 20:10:38.181874 15226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44851 every 8 connection(s)
I20260924 20:10:38.182878 15098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:10:38.185241 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 15098
Sep 24 20:10:38 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:38.195600 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.185889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.0.115.190:39215)
0924 20:10:38.186700 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.186707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.186721 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:10:38.186771 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.186774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.186823 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.186915 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:38.187751 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.188622 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.189835 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.190143 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.190247 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.191926 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:38.191930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:38.191933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:38.192012 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:38.193744 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:38.194227 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:38.194233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:38.194235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:38.194329 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:38.194662 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:38.194664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:38.194665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:38.194829 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:38.195076 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.195239 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.195417 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":694,"spinlock_wait_cycles":896,"thread_start_us":65,"threads_started":1}
I20260924 20:10:38.198925 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.184773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:48187 (local address 127.0.115.190:39215)
0924 20:10:38.184919 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.184923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.185827 (+ 904us) server_negotiation.cc:407] Connection header received
0924 20:10:38.187547 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.187551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.187605 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.187703 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:38.189184 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.189900 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.191391 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.191767 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.191889 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.194821 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:38.194824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:38.194826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:38.194847 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:38.196565 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:38.197293 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:38.197295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:38.197296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:38.197368 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:38.197777 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:38.197779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:38.197780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:38.197994 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:38.198072 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.198717 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.198814 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":8960}
I20260924 20:10:38.199627 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.185174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:48187)
0924 20:10:38.185678 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:38.185731 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:38.187298 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:38.187918 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:38.187927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:38.188406 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:38.189025 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.189040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.190033 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.190037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.191257 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.191267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.191966 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.191972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.192181 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.192750 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:38.192771 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:38.194643 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:38.196785 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:38.196790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:38.196802 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:38.197201 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:38.197460 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:38.197463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:38.197465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:38.197683 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:38.198092 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:38.198098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:38.198331 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 20:10:38.198493 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 20:10:38.198767 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":152,"threads_started":1}
I20260924 20:10:38.200815 15227 heartbeater.cc:347] Connected to a master server at 127.0.115.190:39215
I20260924 20:10:38.201069 15227 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:38.201570 15227 heartbeater.cc:511] Master 127.0.115.190:39215 requested a full tablet report, sending...
I20260924 20:10:38.202623 14769 ts_manager.cc:194] Registered new tserver with Master: 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851)
I20260924 20:10:38.203225 14769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:48187
I20260924 20:10:38.209646 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:10:38.213850 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.210270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.0.115.190:39215)
0924 20:10:38.210392 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.210396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.210558 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:10:38.210700 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.210702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.210828 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.210914 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:10:38.211146 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.211655 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.212529 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.212756 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.212846 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.213133 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:10:38.213543 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:10:38.213641 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.213693 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.213745 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:10:38.214475 14769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:10:38.217927 14769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:10:38.226071 14891 tablet_service.cc:1467] Processing CreateTablet for tablet bd7eb48a26444b4b9685e71753077691 (DEFAULT_TABLE table=test-table [id=16b9f5cc8edf4847ab636f120c1f2784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:38.226791 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7eb48a26444b4b9685e71753077691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:38.229970 14849 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Bootstrap starting.
I20260924 20:10:38.230859 14849 tablet_bootstrap.cc:654] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:38.231112 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.224812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35883 (local address 127.0.0.1:44602)
0924 20:10:38.224943 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:38.224959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:38.225039 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:38.226094 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:38.226097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:38.226115 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:38.226235 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.226245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.227722 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.227726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.229052 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.229063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.230554 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.230557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.230706 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.230744 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 20:10:38.230826 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:10:38.230943 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":29}
I20260924 20:10:38.231356 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.224996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.0.115.130:35883)
0924 20:10:38.225645 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.225659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.225684 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:10:38.225762 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.225767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.225942 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.226109 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:38.226373 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.227588 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.229188 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.230423 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.230533 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.230672 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.231072 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.231184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":203,"threads_started":1}
I20260924 20:10:38.231436 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.225123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44851 (local address 127.0.0.1:37624)
0924 20:10:38.225490 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:38.225504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:38.225588 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:38.226283 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:38.226289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:38.226311 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:38.226394 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.226399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.228146 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.228149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.229198 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.229208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.230519 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.230522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.230619 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.230728 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 20:10:38.230832 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 20:10:38.230970 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":78,"thread_start_us":132,"threads_started":1}
I20260924 20:10:38.231652 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.225352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.0.115.131:44851)
0924 20:10:38.225818 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.225829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.225852 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:10:38.225927 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.225932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.226131 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.226291 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:38.226686 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.227995 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.229359 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.230198 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.230303 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.230451 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.230871 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.231062 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":156,"threads_started":1}
I20260924 20:10:38.232681 14849 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: No bootstrap required, opened a new log
I20260924 20:10:38.232766 14849 ts_tablet_manager.cc:1397] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:10:38.233204 14849 raft_consensus.cc:348] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.233312 14849 raft_consensus.cc:374] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:38.233347 14849 raft_consensus.cc:729] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e82e0c0404068afef23682fe029b5, State: Initialized, Role: FOLLOWER
I20260924 20:10:38.233569 14849 consensus_queue.cc:260] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.234437 14849 ts_tablet_manager.cc:1428] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:10:38.234436 15029 tablet_service.cc:1467] Processing CreateTablet for tablet bd7eb48a26444b4b9685e71753077691 (DEFAULT_TABLE table=test-table [id=16b9f5cc8edf4847ab636f120c1f2784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:38.234359 15161 tablet_service.cc:1467] Processing CreateTablet for tablet bd7eb48a26444b4b9685e71753077691 (DEFAULT_TABLE table=test-table [id=16b9f5cc8edf4847ab636f120c1f2784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:38.235432 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7eb48a26444b4b9685e71753077691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:38.235469 15161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7eb48a26444b4b9685e71753077691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:38.242202 15241 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Bootstrap starting.
I20260924 20:10:38.242208 15240 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Bootstrap starting.
I20260924 20:10:38.244601 15240 tablet_bootstrap.cc:654] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:38.244652 15241 tablet_bootstrap.cc:654] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:38.245532 15240 log.cc:824] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:38.245630 15241 log.cc:824] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:38.247409 15241 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: No bootstrap required, opened a new log
I20260924 20:10:38.247596 15241 ts_tablet_manager.cc:1397] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:10:38.247695 15240 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: No bootstrap required, opened a new log
I20260924 20:10:38.247874 15240 ts_tablet_manager.cc:1397] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:10:38.250516 15241 raft_consensus.cc:348] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.250568 15240 raft_consensus.cc:348] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.250819 15241 raft_consensus.cc:374] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:38.250855 15240 raft_consensus.cc:374] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:38.250881 15241 raft_consensus.cc:729] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f667ab687704df5b37cbb7f99cc61d5, State: Initialized, Role: FOLLOWER
I20260924 20:10:38.250911 15240 raft_consensus.cc:729] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4c2143966e41f991d4983a7f44261e, State: Initialized, Role: FOLLOWER
I20260924 20:10:38.251329 15241 consensus_queue.cc:260] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.251338 15240 consensus_queue.cc:260] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.251978 15227 heartbeater.cc:503] Master 127.0.115.190:39215 was elected leader, sending a full tablet report...
I20260924 20:10:38.251995 15095 heartbeater.cc:503] Master 127.0.115.190:39215 was elected leader, sending a full tablet report...
I20260924 20:10:38.252353 15240 ts_tablet_manager.cc:1428] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 20:10:38.253371 15241 ts_tablet_manager.cc:1428] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 20:10:38.273756 14958 tablet.cc:2369] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:10:38.433912 15228 tablet.cc:2369] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:10:38.484666 15096 tablet.cc:2369] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:38.531160 15245 raft_consensus.cc:482] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:38.531391 15245 raft_consensus.cc:504] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.532841 15245 leader_election.cc:288] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851), d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537)
I20260924 20:10:38.537002 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.533230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:57883 (local address 127.0.115.131:44851)
0924 20:10:38.533363 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.533369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.533563 (+ 194us) server_negotiation.cc:407] Connection header received
0924 20:10:38.533738 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.533742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.533799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.533856 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:38.534207 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.534757 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.535904 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.536423 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.536522 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.536564 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.536796 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.536870 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:10:38.537019 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.533118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44851 (local address 127.0.115.130:57883)
0924 20:10:38.533452 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:38.533469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:38.533610 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:38.533898 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:38.533901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:38.533940 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:38.534080 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.534088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.534879 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.534885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.535776 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.535785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.536546 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.536550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.536652 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.536691 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 20:10:38.536766 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:10:38.536839 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":71,"threads_started":1}
I20260924 20:10:38.537220 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.533134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:42537 (local address 127.0.115.130:53421)
0924 20:10:38.533497 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:38.533516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:38.533650 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:38.534023 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:38.534025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:38.534040 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:38.534160 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.534168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.534931 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.534934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.535786 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:38.535795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.536787 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.536790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:38.536901 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.536923 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:10:38.537000 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:10:38.537081 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":47,"thread_start_us":77,"threads_started":1}
I20260924 20:10:38.537631 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.533228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:53421 (local address 127.0.115.129:42537)
0924 20:10:38.533555 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.533559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.533599 (+ 40us) server_negotiation.cc:407] Connection header received
0924 20:10:38.533784 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.533786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.533864 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.533986 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:38.534284 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.534789 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.535982 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.536643 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.536778 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.536930 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.537014 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.537516 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260924 20:10:38.537631 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7eb48a26444b4b9685e71753077691" candidate_uuid: "ff4c2143966e41f991d4983a7f44261e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f667ab687704df5b37cbb7f99cc61d5" is_pre_election: true
I20260924 20:10:38.537967 15181 raft_consensus.cc:2454] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4c2143966e41f991d4983a7f44261e in term 0.
I20260924 20:10:38.538120 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7eb48a26444b4b9685e71753077691" candidate_uuid: "ff4c2143966e41f991d4983a7f44261e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d10e82e0c0404068afef23682fe029b5" is_pre_election: true
I20260924 20:10:38.538425 14911 raft_consensus.cc:2454] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4c2143966e41f991d4983a7f44261e in term 0.
I20260924 20:10:38.538475 14983 leader_election.cc:302] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f667ab687704df5b37cbb7f99cc61d5, ff4c2143966e41f991d4983a7f44261e; no voters:
I20260924 20:10:38.538786 15245 raft_consensus.cc:2781] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:10:38.538869 15245 raft_consensus.cc:482] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:10:38.538901 15245 raft_consensus.cc:3037] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:38.539995 15245 raft_consensus.cc:504] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.540427 15245 leader_election.cc:288] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [CANDIDATE]: Term 1 election: Requested vote from peers 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851), d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537)
I20260924 20:10:38.540783 15181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7eb48a26444b4b9685e71753077691" candidate_uuid: "ff4c2143966e41f991d4983a7f44261e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f667ab687704df5b37cbb7f99cc61d5"
I20260924 20:10:38.540946 15181 raft_consensus.cc:3037] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:38.540968 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7eb48a26444b4b9685e71753077691" candidate_uuid: "ff4c2143966e41f991d4983a7f44261e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d10e82e0c0404068afef23682fe029b5"
I20260924 20:10:38.541093 14911 raft_consensus.cc:3037] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:38.542297 14911 raft_consensus.cc:2454] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4c2143966e41f991d4983a7f44261e in term 1.
I20260924 20:10:38.542297 15181 raft_consensus.cc:2454] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4c2143966e41f991d4983a7f44261e in term 1.
I20260924 20:10:38.542593 14983 leader_election.cc:302] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f667ab687704df5b37cbb7f99cc61d5, ff4c2143966e41f991d4983a7f44261e; no voters:
I20260924 20:10:38.542843 15245 raft_consensus.cc:2781] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:38.543136 15245 raft_consensus.cc:686] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 1 LEADER]: Becoming Leader. State: Replica: ff4c2143966e41f991d4983a7f44261e, State: Running, Role: LEADER
I20260924 20:10:38.543491 15245 consensus_queue.cc:237] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:38.548313 14768 catalog_manager.cc:5495] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e reported cstate change: term changed from 0 to 1, leader changed from <none> to ff4c2143966e41f991d4983a7f44261e (127.0.115.130). New cstate: current_term: 1 leader_uuid: "ff4c2143966e41f991d4983a7f44261e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } health_report { overall_health: UNKNOWN } } }
I20260924 20:10:38.598518 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.594711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.0.115.130:35883)
0924 20:10:38.594892 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:38.594898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:38.594914 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:10:38.595027 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:38.595029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:38.595086 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:38.595208 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:10:38.595451 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.596116 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.596971 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:38.597278 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:38.597386 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:38.597708 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:10:38.598136 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:10:38.598275 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:38.598316 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 20:10:38.598371 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:10:38.604559 15181 raft_consensus.cc:1261] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 FOLLOWER]: Refusing update from remote peer ff4c2143966e41f991d4983a7f44261e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:38.604555 14911 raft_consensus.cc:1261] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Refusing update from remote peer ff4c2143966e41f991d4983a7f44261e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:38.605304 15249 consensus_queue.cc:1035] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:38.605501 15245 consensus_queue.cc:1035] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:38.611461 15253 mvcc.cc:205] Tried to move back new op lower bound from 7332989495716548608 to 7332989495477563392. Current Snapshot: MvccSnapshot[applied={T|T < 7332989495716548608}]
I20260924 20:10:38.613202 15254 mvcc.cc:205] Tried to move back new op lower bound from 7332989495716548608 to 7332989495477563392. Current Snapshot: MvccSnapshot[applied={T|T < 7332989495716548608}]
I20260924 20:10:38.621359 14769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:38.621542 14769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:38.623800 14769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Sending DeleteTablet for 3 replicas of tablet bd7eb48a26444b4b9685e71753077691
I20260924 20:10:38.624492 15029 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7eb48a26444b4b9685e71753077691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:38 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:44602
I20260924 20:10:38.624672 15161 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7eb48a26444b4b9685e71753077691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:38 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:37624
I20260924 20:10:38.624833 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14807
I20260924 20:10:38.625691 15263 tablet_replica.cc:330] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: stopping tablet replica
I20260924 20:10:38.626030 15263 raft_consensus.cc:2229] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:10:38.626331 15263 raft_consensus.cc:2258] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:38.626468 15262 tablet_replica.cc:330] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: stopping tablet replica
I20260924 20:10:38.626744 15262 raft_consensus.cc:2229] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:10:38.627171 15262 raft_consensus.cc:2258] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:38.628364 15263 ts_tablet_manager.cc:1905] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:38.629278 15262 ts_tablet_manager.cc:1905] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:38.631771 15263 ts_tablet_manager.cc:1918] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:38.631888 15263 log.cc:1196] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/bd7eb48a26444b4b9685e71753077691
I20260924 20:10:38.631922 15262 ts_tablet_manager.cc:1918] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:38.632179 15262 log.cc:1196] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/bd7eb48a26444b4b9685e71753077691
I20260924 20:10:38.632267 15263 ts_tablet_manager.cc:1939] T bd7eb48a26444b4b9685e71753077691 P 7f667ab687704df5b37cbb7f99cc61d5: Deleting consensus metadata
I20260924 20:10:38.632557 15262 ts_tablet_manager.cc:1939] T bd7eb48a26444b4b9685e71753077691 P ff4c2143966e41f991d4983a7f44261e: Deleting consensus metadata
W20260924 20:10:38.633598 14753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:42537 (error 108)
I20260924 20:10:38.633805 14756 catalog_manager.cc:4841] TS ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883): tablet bd7eb48a26444b4b9685e71753077691 (table test-table [id=16b9f5cc8edf4847ab636f120c1f2784]) successfully deleted
I20260924 20:10:38.634132 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14962
I20260924 20:10:38.634343 14754 catalog_manager.cc:4841] TS 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851): tablet bd7eb48a26444b4b9685e71753077691 (table test-table [id=16b9f5cc8edf4847ab636f120c1f2784]) successfully deleted
I20260924 20:10:38.635548 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:38.635181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:42537 (local address 127.0.0.1:51362)
0924 20:10:38.635357 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:38.635437 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 20:10:38.635715 14753 catalog_manager.cc:4584] TS d10e82e0c0404068afef23682fe029b5 (127.0.115.129:42537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd7eb48a26444b4b9685e71753077691: Network error: Client connection negotiation failed: client connection to 127.0.115.129:42537: connect: Connection refused (error 111)
I20260924 20:10:38.641527 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 15098
I20260924 20:10:38.647360 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 14736
I20260924 20:10:38.653843 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39215
--webserver_interface=127.0.115.190
--webserver_port=45199
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:39215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:38.763504 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:38.763754 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:38.763792 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:38.767621 15265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:38.767669 15265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:38.767688 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:38.767704 15265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:38.767719 15265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:38.772864 15265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:39215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:39215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:38.773828 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:38.774762 15265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:38.783245 15273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:38.783245 15271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:38.783263 15270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:38.783601 15265 server_base.cc:1034] running on GCE node
I20260924 20:10:38.784333 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:38.785403 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:38.786624 15265 hybrid_clock.cc:648] HybridClock initialized: now 1790280638786607 us; error 35 us; skew 500 ppm
Sep 24 20:10:38 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:38.789347 15265 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:38.790259 15265 webserver.cc:460] Webserver started at http://127.0.115.190:45199/ using document root <none> and password file <none>
I20260924 20:10:38.790817 15265 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:38.790905 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:38.794646 15265 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:10:38.796746 15280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:38.797875 15265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:10:38.798024 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "268cfa47ace14690a4f74581462e4c27"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "ba43dd26d6a3d3b0a83bdb767d5ef754"
server_key_iv: "6100d83dde379855260b5812d04b250e"
server_key_version: "encryptionkey@0"
I20260924 20:10:38.798576 15265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:38.808118 15265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:38.810845 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:38.811057 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:38.818079 15265 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:39215
I20260924 20:10:38.818360 15332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:39215 every 8 connection(s)
I20260924 20:10:38.819180 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:38.819857 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 15265
I20260924 20:10:38.820297 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:42537
--local_ip_for_outbound_sockets=127.0.115.129
--tserver_master_addrs=127.0.115.190:39215
--webserver_port=40375
--webserver_interface=127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:10:38.823645 15333 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:10:38.825318 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Bootstrap starting.
I20260924 20:10:38.842994 15333 log.cc:824] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:38.854364 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:10:38.854777 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27: Bootstrap complete.
I20260924 20:10:38.858903 15333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:38.859303 15333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268cfa47ace14690a4f74581462e4c27, State: Initialized, Role: FOLLOWER
I20260924 20:10:38.859859 15333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:38.860041 15333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:10:38.860139 15333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:10:38.860258 15333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:10:38.862155 15333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:38.862495 15333 leader_election.cc:302] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268cfa47ace14690a4f74581462e4c27; no voters:
I20260924 20:10:38.862866 15333 leader_election.cc:288] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 20:10:38.862957 15337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:10:38.863179 15337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [term 3 LEADER]: Becoming Leader. State: Replica: 268cfa47ace14690a4f74581462e4c27, State: Running, Role: LEADER
I20260924 20:10:38.863530 15337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } }
I20260924 20:10:38.864167 15333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:38.865022 15339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268cfa47ace14690a4f74581462e4c27. Latest consensus state: current_term: 3 leader_uuid: "268cfa47ace14690a4f74581462e4c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } } }
I20260924 20:10:38.865157 15339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:38.865033 15338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "268cfa47ace14690a4f74581462e4c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cfa47ace14690a4f74581462e4c27" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 39215 } } }
I20260924 20:10:38.865346 15338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cfa47ace14690a4f74581462e4c27 [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:38.865823 15345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:38.871084 15345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:38.871810 15345 catalog_manager.cc:1259] Loaded cluster ID: 28b86c4a24af4ae29ba3b0f268af6479
I20260924 20:10:38.871894 15345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 20:10:38.872756 15345 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa2c2f049ad google::LogMessage::Fail() at ??:0
@ 0x7fa2c2f0788a google::LogMessage::SendToLog() at ??:0
@ 0x7fa2c2f044de google::LogMessage::Flush() at ??:0
@ 0x7fa2c2f05a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa2cd847fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa2cd848d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa2cd845579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa2cd889640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa2c51563d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa2c5156c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa2c51583f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa2c5147512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa2c678b6db start_thread at ??:0
@ 0x7fa2c1a0261f clone at ??:0
W20260924 20:10:38.963857 15335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:38.964195 15335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:38.964269 15335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:38.968372 15335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:38.968427 15335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:38.968487 15335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:10:38.973845 15335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:42537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=40375
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:38.974957 15335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:38.976050 15335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:38.985873 15375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:38.985883 15374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:38.986172 15335 server_base.cc:1034] running on GCE node
W20260924 20:10:38.986366 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:38.986811 15335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:38.987560 15335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:38.988760 15335 hybrid_clock.cc:648] HybridClock initialized: now 1790280638988728 us; error 46 us; skew 500 ppm
Sep 24 20:10:38 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:38.991874 15335 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:10:38.992841 15335 webserver.cc:460] Webserver started at http://127.0.115.129:40375/ using document root <none> and password file <none>
I20260924 20:10:38.993456 15335 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:38.993558 15335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:38.997602 15335 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:10:38.999884 15384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:39.001220 15335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:39.001360 15335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "d10e82e0c0404068afef23682fe029b5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "3032052a9c696b56060bb19a2692d034"
server_key_iv: "cfd4a2634749ce75f60a003882df9332"
server_key_version: "encryptionkey@0"
I20260924 20:10:39.001945 15335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:39.028645 15335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:39.031893 15335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:39.032127 15335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:39.032737 15335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:39.034010 15391 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:10:39.039467 15335 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:10:39.039558 15335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 20:10:39.039644 15335 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:10:39.041419 15335 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:10:39.041489 15335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 20:10:39.041812 15391 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Bootstrap starting.
I20260924 20:10:39.052316 15335 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:42537
I20260924 20:10:39.052376 15498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:42537 every 8 connection(s)
I20260924 20:10:39.053318 15335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:10:39.057744 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 15335
I20260924 20:10:39.058224 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:35883
--local_ip_for_outbound_sockets=127.0.115.130
--tserver_master_addrs=127.0.115.190:39215
--webserver_port=35613
--webserver_interface=127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:10:39.063243 15391 log.cc:824] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:39.070549 15391 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:10:39.070936 15391 tablet_bootstrap.cc:492] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Bootstrap complete.
I20260924 20:10:39.072142 15391 ts_tablet_manager.cc:1397] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.021s
I20260924 20:10:39.074658 15391 raft_consensus.cc:348] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:39.075109 15391 raft_consensus.cc:729] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e82e0c0404068afef23682fe029b5, State: Initialized, Role: FOLLOWER
I20260924 20:10:39.075592 15391 consensus_queue.cc:260] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:39.076565 15391 ts_tablet_manager.cc:1428] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
W20260924 20:10:39.151655 15501 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:10:39.055280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:39791)
0924 20:10:39.055788 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.055842 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:39.056748 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:39.150885 (+ 94137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":160,"threads_started":1}
W20260924 20:10:39.152323 15499 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:39215 (0 consecutive failures): Network error: Failed to ping master at 127.0.115.190:39215: Client connection negotiation failed: client connection to 127.0.115.190:39215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:10:39.153080 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.152824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:46991)
0924 20:10:39.152951 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.153021 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 20:10:39.153797 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.153606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:53831)
0924 20:10:39.153707 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.153750 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 20:10:39.154054 15499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:10:39.196560 15502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:39.196796 15502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:39.196838 15502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:39.200847 15502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:39.200902 15502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:39.201025 15502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:10:39.206351 15502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=35613
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:39.207342 15502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:39.208424 15502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:39.217849 15512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:39.217849 15510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:39.217875 15509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:39.218277 15502 server_base.cc:1034] running on GCE node
I20260924 20:10:39.218696 15502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:39.219331 15502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:39.220520 15502 hybrid_clock.cc:648] HybridClock initialized: now 1790280639220502 us; error 29 us; skew 500 ppm
Sep 24 20:10:39 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:39.223423 15502 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:10:39.224385 15502 webserver.cc:460] Webserver started at http://127.0.115.130:35613/ using document root <none> and password file <none>
I20260924 20:10:39.224963 15502 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:39.225062 15502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:39.229167 15502 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:10:39.231156 15519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:39.232466 15502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:39.232606 15502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "ff4c2143966e41f991d4983a7f44261e"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "c66d91524fb6481434bdf635beccf598"
server_key_iv: "d8d71afe7ed06c91602e54b95c5229cc"
server_key_version: "encryptionkey@0"
I20260924 20:10:39.233146 15502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:39.261008 15502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:39.263816 15502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:39.264038 15502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:39.264657 15502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:39.265486 15502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:39.265563 15502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:39.265636 15502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:39.265688 15502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:39.274536 15502 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:35883
I20260924 20:10:39.274556 15632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:35883 every 8 connection(s)
I20260924 20:10:39.275540 15502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:10:39.277024 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 15502
I20260924 20:10:39.277417 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:44851
--local_ip_for_outbound_sockets=127.0.115.131
--tserver_master_addrs=127.0.115.190:39215
--webserver_port=46181
--webserver_interface=127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:10:39.279901 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.277779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:45079)
0924 20:10:39.278296 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.278443 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
W20260924 20:10:39.280390 15633 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:39215 (0 consecutive failures): Network error: Failed to ping master at 127.0.115.190:39215: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
I20260924 20:10:39.281050 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.280750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:54633)
0924 20:10:39.280900 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.280965 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 20:10:39.281973 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.281716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:57371)
0924 20:10:39.281828 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.281913 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 20:10:39.282204 15633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:10:39.390309 15636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:39.390612 15636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:39.390654 15636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:39.394600 15636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:39.394655 15636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:39.394711 15636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:10:39.400043 15636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:44851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=46181
--tserver_master_addrs=127.0.115.190:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:39.401110 15636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:39.402073 15636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:39.412113 15644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:39.412114 15642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:39.412150 15641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:39.412462 15636 server_base.cc:1034] running on GCE node
I20260924 20:10:39.412941 15636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:39.413668 15636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:39.414857 15636 hybrid_clock.cc:648] HybridClock initialized: now 1790280639414821 us; error 45 us; skew 500 ppm
Sep 24 20:10:39 dist-test-slave-2pm7 krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:39.417629 15636 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:10:39.418470 15636 webserver.cc:460] Webserver started at http://127.0.115.131:46181/ using document root <none> and password file <none>
I20260924 20:10:39.418984 15636 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:39.419050 15636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:39.422834 15636 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:10:39.424968 15651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:39.426247 15636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:10:39.426352 15636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "7f667ab687704df5b37cbb7f99cc61d5"
format_stamp: "Formatted at 2026-09-24 20:10:36 on dist-test-slave-2pm7"
server_key: "8319b706cea9bdf8545f6175c85377c9"
server_key_iv: "bb93fc44aa733934ddc49cc78cc58a23"
server_key_version: "encryptionkey@0"
I20260924 20:10:39.426880 15636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:39.462610 15636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:39.466030 15636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:39.466193 15636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:39.466751 15636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:39.467552 15636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:39.467599 15636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:39.467641 15636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:39.467664 15636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:39.476435 15636 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44851
I20260924 20:10:39.476464 15764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44851 every 8 connection(s)
I20260924 20:10:39.477433 15636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:10:39.481328 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.479541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:55799)
0924 20:10:39.480067 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.480178 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
W20260924 20:10:39.481930 15765 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:39215 (0 consecutive failures): Network error: Failed to ping master at 127.0.115.190:39215: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
I20260924 20:10:39.482807 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.482383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:35529)
0924 20:10:39.482635 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.482711 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 20:10:39.483490 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:39.483262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:37981)
0924 20:10:39.483380 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:39.483428 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:10:39.483800 15765 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:10:39.484673 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 15636
I20260924 20:10:40.155160 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:40.154723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:42937)
0924 20:10:40.154989 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:40.155053 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20260924 20:10:40.283119 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:40.282737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:50199)
0924 20:10:40.282963 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:40.283016 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260924 20:10:40.484845 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:40.484419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:54407)
0924 20:10:40.484684 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:40.484742 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260924 20:10:40.786350 15772 raft_consensus.cc:482] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:40.786532 15772 raft_consensus.cc:504] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:40.787613 15772 leader_election.cc:288] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883), 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851)
Sep 24 20:10:40 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.131@KRBTEST.COM
Sep 24 20:10:40 dist-test-slave-2pm7 krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.130@KRBTEST.COM
I20260924 20:10:40.801676 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:40.787868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44851 (local address 127.0.115.129:41557)
0924 20:10:40.788294 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:40.788307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:40.788375 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:40.790131 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:40.790134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:40.790665 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:40.791149 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:40.791164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.792516 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.792522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:40.793399 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:40.793410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.794165 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.794169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:40.794396 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:40.795104 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:40.795130 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:40.796856 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:40.798952 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:40.798968 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:40.798983 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:40.799321 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:40.799777 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:40.799780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:40.799782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:40.799946 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:40.800678 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:40.800684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:40.800945 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 20:10:40.801081 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 20:10:40.801429 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":133,"thread_start_us":53,"threads_started":1}
I20260924 20:10:40.801687 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:40.788059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:38233 (local address 127.0.115.130:35883)
0924 20:10:40.788504 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:40.788518 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:40.788558 (+ 40us) server_negotiation.cc:407] Connection header received
0924 20:10:40.789248 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:40.789255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:40.789615 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:40.789872 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:40.791334 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.792366 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.793552 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.794028 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.794207 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:40.797368 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:40.797372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:40.797383 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:40.797410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:40.799326 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:40.799800 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:40.799805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:40.799818 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:40.799980 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:40.800271 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:40.800273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:40.800274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:40.800725 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:40.800846 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:40.801258 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 20:10:40.801520 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":96,"threads_started":1}
I20260924 20:10:40.801726 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:40.788076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:41557 (local address 127.0.115.131:44851)
0924 20:10:40.788631 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:40.788649 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:40.788683 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:10:40.789394 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:40.789400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:40.789751 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:40.790156 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:40.791328 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.792366 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.793601 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.794039 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.794208 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:40.797112 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:40.797120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:40.797139 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:40.797178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:40.798828 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:40.799451 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:40.799457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:40.799462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:40.799642 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:40.800082 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:40.800085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:40.800087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:40.800532 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:40.800663 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:40.801251 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 20:10:40.801523 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":102,"threads_started":1}
I20260924 20:10:40.801714 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:40.787868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35883 (local address 127.0.115.129:38233)
0924 20:10:40.788220 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:40.788235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:40.788362 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:40.790079 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:40.790085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:40.790670 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:40.791149 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:40.791164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.792518 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.792521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:40.793401 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:40.793410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:40.794168 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:40.794172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:40.794399 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:40.795167 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:40.795180 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:40.797198 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:40.799440 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:40.799444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:40.799446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:40.799694 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:40.800082 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:40.800085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:40.800087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:40.800178 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:40.800831 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:40.800834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:40.800937 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 20:10:40.801123 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 20:10:40.801523 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":324,"spinlock_wait_cycles":8320,"thread_start_us":79,"threads_started":1}
W20260924 20:10:40.804715 15386 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
W20260924 20:10:40.804997 15388 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
I20260924 20:10:40.805090 15388 leader_election.cc:302] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e82e0c0404068afef23682fe029b5; no voters: 7f667ab687704df5b37cbb7f99cc61d5, ff4c2143966e41f991d4983a7f44261e
I20260924 20:10:40.805367 15772 raft_consensus.cc:2726] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:10:41.156569 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:41.156224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:56751)
0924 20:10:41.156397 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:41.156473 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 20:10:41.284433 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:41.283890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:45209)
0924 20:10:41.284261 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:41.284321 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":173,"threads_started":1}
I20260924 20:10:41.486219 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:41.485703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:48451)
0924 20:10:41.486008 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:41.486098 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260924 20:10:42.157768 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:42.157342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:39329)
0924 20:10:42.157612 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:42.157669 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260924 20:10:42.285543 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:42.285130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:34151)
0924 20:10:42.285404 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:42.285454 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
I20260924 20:10:42.487560 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:42.487084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:36999)
0924 20:10:42.487398 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:42.487462 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260924 20:10:42.819293 15782 raft_consensus.cc:482] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:42.819417 15782 raft_consensus.cc:504] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:42.819798 15782 leader_election.cc:288] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883), 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851)
W20260924 20:10:42.820683 15388 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
W20260924 20:10:42.820781 15386 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
I20260924 20:10:42.820894 15386 leader_election.cc:302] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e82e0c0404068afef23682fe029b5; no voters: 7f667ab687704df5b37cbb7f99cc61d5, ff4c2143966e41f991d4983a7f44261e
I20260924 20:10:42.821055 15782 raft_consensus.cc:2726] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:10:43.159015 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:43.158595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:39975)
0924 20:10:43.158849 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:43.158902 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
I20260924 20:10:43.286772 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:43.286340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:58537)
0924 20:10:43.286595 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:43.286674 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260924 20:10:43.488878 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:43.488438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:46175)
0924 20:10:43.488714 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:43.488783 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260924 20:10:44.160440 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:44.159964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:47309)
0924 20:10:44.160285 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:44.160344 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":70,"threads_started":1}
I20260924 20:10:44.288138 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:44.287672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:48267)
0924 20:10:44.287975 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:44.288047 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260924 20:10:44.490103 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:44.489664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:55341)
0924 20:10:44.489922 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:44.489977 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
I20260924 20:10:44.811129 15789 raft_consensus.cc:482] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:44.811271 15789 raft_consensus.cc:504] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:44.811821 15789 leader_election.cc:288] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883), 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851)
W20260924 20:10:44.812681 15386 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
W20260924 20:10:44.812770 15388 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
I20260924 20:10:44.812879 15388 leader_election.cc:302] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e82e0c0404068afef23682fe029b5; no voters: 7f667ab687704df5b37cbb7f99cc61d5, ff4c2143966e41f991d4983a7f44261e
I20260924 20:10:44.813036 15789 raft_consensus.cc:2726] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:10:45.161841 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:45.161385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:39421)
0924 20:10:45.161674 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:45.161737 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260924 20:10:45.289222 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:45.288822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:53209)
0924 20:10:45.289073 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:45.289128 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260924 20:10:45.491464 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:45.490994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:48761)
0924 20:10:45.491285 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:45.491366 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
I20260924 20:10:46.163113 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:46.162695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:39049)
0924 20:10:46.162957 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:46.163010 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260924 20:10:46.290845 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:46.290219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:44073)
0924 20:10:46.290685 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:46.290748 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":80,"threads_started":1}
I20260924 20:10:46.492918 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:46.492443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:34577)
0924 20:10:46.492762 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:46.492822 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
I20260924 20:10:47.164472 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:47.164055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:45995)
0924 20:10:47.164311 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:47.164369 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260924 20:10:47.292121 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:47.291602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:44403)
0924 20:10:47.291895 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:47.292022 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260924 20:10:47.494241 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:47.493780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:45591)
0924 20:10:47.494062 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:47.494140 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
I20260924 20:10:48.165697 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:48.165243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:35901)
0924 20:10:48.165519 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:48.165586 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20260924 20:10:48.293478 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:48.293016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:54611)
0924 20:10:48.293320 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:48.293381 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260924 20:10:48.495548 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:48.495092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:54093)
0924 20:10:48.495393 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:48.495448 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260924 20:10:48.561306 15802 raft_consensus.cc:482] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:48.561434 15802 raft_consensus.cc:504] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:48.561815 15802 leader_election.cc:288] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883), 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851)
W20260924 20:10:48.562618 15388 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
W20260924 20:10:48.562748 15386 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
I20260924 20:10:48.562850 15386 leader_election.cc:302] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e82e0c0404068afef23682fe029b5; no voters: 7f667ab687704df5b37cbb7f99cc61d5, ff4c2143966e41f991d4983a7f44261e
I20260924 20:10:48.563004 15802 raft_consensus.cc:2726] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:10:49.167052 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:49.166619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:51047)
0924 20:10:49.166892 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:49.166953 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260924 20:10:49.294750 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:49.294327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:56331)
0924 20:10:49.294590 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:49.294645 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260924 20:10:49.497082 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:49.496456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:45473)
0924 20:10:49.496829 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:49.496900 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260924 20:10:50.168344 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:50.167862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:51901)
0924 20:10:50.168190 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:50.168246 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":161,"threads_started":1}
I20260924 20:10:50.295881 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:50.295474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:40857)
0924 20:10:50.295730 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:50.295784 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260924 20:10:50.498333 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:50.497893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:37281)
0924 20:10:50.498163 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:50.498233 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260924 20:10:51.169646 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:51.169184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:37995)
0924 20:10:51.169479 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:51.169546 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260924 20:10:51.297195 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:51.296726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:58401)
0924 20:10:51.297027 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:51.297090 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
I20260924 20:10:51.499717 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:51.499245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:33247)
0924 20:10:51.499557 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:51.499621 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
I20260924 20:10:52.170847 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:52.170457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:33233)
0924 20:10:52.170694 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:52.170749 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260924 20:10:52.298439 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:52.297992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:50967)
0924 20:10:52.298279 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:52.298339 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260924 20:10:52.501137 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:52.500608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:33369)
0924 20:10:52.500923 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:52.501020 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260924 20:10:52.961709 15815 raft_consensus.cc:482] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:52.961905 15815 raft_consensus.cc:504] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4c2143966e41f991d4983a7f44261e" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35883 } } peers { permanent_uuid: "7f667ab687704df5b37cbb7f99cc61d5" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44851 } } peers { permanent_uuid: "d10e82e0c0404068afef23682fe029b5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 42537 } }
I20260924 20:10:52.962345 15815 leader_election.cc:288] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883), 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851)
W20260924 20:10:52.963222 15386 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f667ab687704df5b37cbb7f99cc61d5 (127.0.115.131:44851): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
W20260924 20:10:52.963335 15388 leader_election.cc:341] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff4c2143966e41f991d4983a7f44261e (127.0.115.130:35883): Not found: Tablet not found: bd7eb48a26444b4b9685e71753077691
I20260924 20:10:52.963459 15388 leader_election.cc:302] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d10e82e0c0404068afef23682fe029b5; no voters: 7f667ab687704df5b37cbb7f99cc61d5, ff4c2143966e41f991d4983a7f44261e
I20260924 20:10:52.963609 15815 raft_consensus.cc:2726] T bd7eb48a26444b4b9685e71753077691 P d10e82e0c0404068afef23682fe029b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:10:53.172092 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:53.171598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:54903)
0924 20:10:53.171872 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:53.171932 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260924 20:10:53.299599 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:53.299195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:43539)
0924 20:10:53.299446 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:53.299502 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260924 20:10:53.502485 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:53.502021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.131:59717)
0924 20:10:53.502310 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:53.502389 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260924 20:10:54.173386 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:54.172997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.129:60961)
0924 20:10:54.173240 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:54.173291 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260924 20:10:54.300879 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:54.300488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:39215 (local address 127.0.115.130:37041)
0924 20:10:54.300738 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:54.300785 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260924 20:10:54.485795 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 15335
I20260924 20:10:54.492214 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 15502
I20260924 20:10:54.497859 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 15636
2026-09-24T20:10:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19830 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:10:54 dist-test-slave-2pm7 krb5kdc[15823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:10:54 dist-test-slave-2pm7 krb5kdc[15823](info): set up 2 sockets
Sep 24 20:10:54 dist-test-slave-2pm7 krb5kdc[15823](info): commencing operation
krb5kdc: starting...
W20260924 20:10:55.544397 462 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:10:55 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:10:55.666553 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:43451
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:55.777701 15836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:55.777992 15836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:55.778059 15836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:55.782085 15836 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:10:55.782155 15836 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:10:55.782181 15836 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:10:55.782212 15836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:10:55.782253 15836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:55.782305 15836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:55.782342 15836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:10:55.782379 15836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:10:55.787492 15836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:43451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:55.788635 15836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:55.789705 15836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:10:55.798501 15836 server_base.cc:1034] running on GCE node
W20260924 20:10:55.798439 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:55.798518 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:55.798432 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:55.799281 15836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:55.800469 15836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:55.801630 15836 hybrid_clock.cc:648] HybridClock initialized: now 1790280655801615 us; error 26 us; skew 500 ppm
Sep 24 20:10:55 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:55.804512 15836 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:10:55.805435 15836 webserver.cc:460] Webserver started at http://127.0.115.190:34743/ using document root <none> and password file <none>
I20260924 20:10:55.806028 15836 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:55.806119 15836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:55.806460 15836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:55.808758 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "ba9770723fb049ce916f20faec01529f"
format_stamp: "Formatted at 2026-09-24 20:10:55 on dist-test-slave-2pm7"
server_key: "6cacbc369e2140fc9007f1cf5225b920"
server_key_iv: "f38ef47ce39aa64b740be75e7ff2b714"
server_key_version: "encryptionkey@0"
I20260924 20:10:55.809334 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "ba9770723fb049ce916f20faec01529f"
format_stamp: "Formatted at 2026-09-24 20:10:55 on dist-test-slave-2pm7"
server_key: "6cacbc369e2140fc9007f1cf5225b920"
server_key_iv: "f38ef47ce39aa64b740be75e7ff2b714"
server_key_version: "encryptionkey@0"
I20260924 20:10:55.813126 15836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:10:55.815562 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:55.816882 15836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:10:55.817020 15836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "ba9770723fb049ce916f20faec01529f"
format_stamp: "Formatted at 2026-09-24 20:10:55 on dist-test-slave-2pm7"
server_key: "6cacbc369e2140fc9007f1cf5225b920"
server_key_iv: "f38ef47ce39aa64b740be75e7ff2b714"
server_key_version: "encryptionkey@0"
I20260924 20:10:55.817134 15836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:55.829568 15836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:55.832549 15836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:55.832758 15836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:55.839818 15836 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:43451
I20260924 20:10:55.840112 15903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:43451 every 8 connection(s)
I20260924 20:10:55.840862 15836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:10:55.842388 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 15836
I20260924 20:10:55.842526 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:10:55.842869 462 external_mini_cluster.cc:1428] Setting key 4686961cb40b6ad6ba2ddbe5780f930a
I20260924 20:10:55.844208 15904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:55.850674 15904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Bootstrap starting.
Sep 24 20:10:55 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:55.853343 15904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:55.854326 15904 log.cc:824] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:55.856305 15904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: No bootstrap required, opened a new log
I20260924 20:10:55.857311 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:55.844298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.0.115.190:43451)
0924 20:10:55.844718 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:55.844733 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:55.844775 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:10:55.845428 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:55.845444 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:55.845784 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:55.846061 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:55.846955 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:55.848005 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:55.849135 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:55.849585 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:55.849699 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:55.852473 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:55.852480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:55.852496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:55.852527 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:55.854521 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:55.854969 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:55.854975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:55.854980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:55.855092 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:55.855364 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:55.855368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:55.855370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:55.855755 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:55.855983 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:55.856280 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 20:10:55.856604 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":132,"threads_started":1}
I20260924 20:10:55.859520 15904 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:10:55.859791 15904 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:55.859840 15904 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9770723fb049ce916f20faec01529f, State: Initialized, Role: FOLLOWER
I20260924 20:10:55.860342 15904 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:10:55.860481 15904 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:10:55.860525 15904 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:10:55.860597 15904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:55.861434 15904 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:10:55.861723 15904 leader_election.cc:302] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba9770723fb049ce916f20faec01529f; no voters:
I20260924 20:10:55.861966 15904 leader_election.cc:288] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:10:55.862088 15909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:55.862319 15909 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 LEADER]: Becoming Leader. State: Replica: ba9770723fb049ce916f20faec01529f, State: Running, Role: LEADER
I20260924 20:10:55.862689 15909 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:10:55.863225 15904 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:10:55.864341 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba9770723fb049ce916f20faec01529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } } }
I20260924 20:10:55.864476 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:55.864854 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9770723fb049ce916f20faec01529f. Latest consensus state: current_term: 1 leader_uuid: "ba9770723fb049ce916f20faec01529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } } }
I20260924 20:10:55.864898 15918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:10:55.864981 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: This master's current role is: LEADER
I20260924 20:10:55.867646 15918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:10:55.873860 15918 catalog_manager.cc:1347] Generated new cluster ID: ef8648ae908e4845a1fbd3eeb04aa781
I20260924 20:10:55.873939 15918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:10:55.902841 15918 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:10:55.904042 15918 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:10:55.910446 15918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Generated new TSK 0
I20260924 20:10:55.911262 15918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:10:55.951086 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:43451
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:56.065457 15930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:56.065762 15930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:56.065838 15930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:56.070031 15930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:56.070113 15930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:56.070209 15930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:10:56.075515 15930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:56.076691 15930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:56.077744 15930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:56.087442 15936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:56.087586 15930 server_base.cc:1034] running on GCE node
W20260924 20:10:56.087464 15935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:56.087456 15938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:56.088140 15930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:56.088729 15930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:56.089895 15930 hybrid_clock.cc:648] HybridClock initialized: now 1790280656089877 us; error 27 us; skew 500 ppm
Sep 24 20:10:56 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:56.092643 15930 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:10:56.093457 15930 webserver.cc:460] Webserver started at http://127.0.115.129:38063/ using document root <none> and password file <none>
I20260924 20:10:56.094022 15930 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:56.094084 15930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:56.094324 15930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:56.096446 15930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "84bcf6925e1b4e99ae523c7de879803c"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "116d7c88e3859e61ff837ab3b6e049e2"
server_key_iv: "075c8610c300f61be36ec91ec00781d9"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.096997 15930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "84bcf6925e1b4e99ae523c7de879803c"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "116d7c88e3859e61ff837ab3b6e049e2"
server_key_iv: "075c8610c300f61be36ec91ec00781d9"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.100554 15930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:10:56.102916 15945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.104182 15930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 20:10:56.104313 15930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "84bcf6925e1b4e99ae523c7de879803c"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "116d7c88e3859e61ff837ab3b6e049e2"
server_key_iv: "075c8610c300f61be36ec91ec00781d9"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.104411 15930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:56.127729 15930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:56.130697 15930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:56.130893 15930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:56.131513 15930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:56.132421 15930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:56.132467 15930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.132534 15930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:56.132565 15930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.141549 15930 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:40017
I20260924 20:10:56.141575 16058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:40017 every 8 connection(s)
I20260924 20:10:56.142449 15930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:10:56.148082 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 15930
I20260924 20:10:56.148218 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:10:56.148515 462 external_mini_cluster.cc:1428] Setting key 3b4756a2c9afb44bd5a950999cca63c8
Sep 24 20:10:56 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:56.157352 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.144365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:38703 (local address 127.0.115.190:43451)
0924 20:10:56.144536 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.144541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.145607 (+ 1066us) server_negotiation.cc:407] Connection header received
0924 20:10:56.146489 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.146492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.146539 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.146612 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:56.147707 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.148572 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.150177 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.150532 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.150726 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.153433 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:56.153437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:56.153439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:56.153469 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:56.154966 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.155607 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.155610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.155612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.155699 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.156149 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.156153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.156154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.156342 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:56.156429 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.157111 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.157227 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:10:56.158604 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.144759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:38703)
0924 20:10:56.145462 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.145504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.146314 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.146755 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.146761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.147113 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:56.147585 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.147598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.148731 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.148735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.150047 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.150058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.150716 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.150720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.150863 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.151485 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:56.151524 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:56.153270 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:56.155088 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:56.155097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:56.155116 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:56.155501 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:56.155802 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:56.155807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:56.155811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:56.156047 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:56.156454 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:56.156463 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:56.156808 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.157043 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.157446 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:10:56.159890 16059 heartbeater.cc:347] Connected to a master server at 127.0.115.190:43451
I20260924 20:10:56.160245 16059 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:56.160768 16059 heartbeater.cc:511] Master 127.0.115.190:43451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:10:56.162555 15868 ts_manager.cc:194] Registered new tserver with Master: 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017)
I20260924 20:10:56.164131 15868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:38703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:10:56.178542 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:43451
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:56.288882 16064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:56.289120 16064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:56.289162 16064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:56.293007 16064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:56.293051 16064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:56.293104 16064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:10:56.298341 16064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:56.299523 16064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:56.300521 16064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:56.310099 16072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:56.310102 16070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:56.310206 16069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:56.310355 16064 server_base.cc:1034] running on GCE node
I20260924 20:10:56.310774 16064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:56.311357 16064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:56.312536 16064 hybrid_clock.cc:648] HybridClock initialized: now 1790280656312510 us; error 37 us; skew 500 ppm
Sep 24 20:10:56 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:56.315220 16064 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:10:56.316084 16064 webserver.cc:460] Webserver started at http://127.0.115.130:39655/ using document root <none> and password file <none>
I20260924 20:10:56.316617 16064 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:56.316673 16064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:56.316835 16064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:56.318856 16064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "9b025e00fa64475f8a2c69cbc8bc357f"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "85c76b031d1c6a2115030ac7d5542034"
server_key_iv: "e013cf0c319ca9118bbdfc331245a0c3"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.319379 16064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "9b025e00fa64475f8a2c69cbc8bc357f"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "85c76b031d1c6a2115030ac7d5542034"
server_key_iv: "e013cf0c319ca9118bbdfc331245a0c3"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.323386 16064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 20:10:56.325822 16079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.327028 16064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:10:56.327159 16064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "9b025e00fa64475f8a2c69cbc8bc357f"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "85c76b031d1c6a2115030ac7d5542034"
server_key_iv: "e013cf0c319ca9118bbdfc331245a0c3"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.327283 16064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:56.356849 16064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:56.360244 16064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:56.360451 16064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:56.361060 16064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:56.361893 16064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:56.361938 16064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.362017 16064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:56.362051 16064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.370700 16064 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:35647
I20260924 20:10:56.370783 16192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:35647 every 8 connection(s)
I20260924 20:10:56.371606 16064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:10:56.374877 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16064
I20260924 20:10:56.374972 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:10:56.375270 462 external_mini_cluster.cc:1428] Setting key afed41293736400b3f2920edff7e0a1e
Sep 24 20:10:56 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:10:56.386720 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.373433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:37937 (local address 127.0.115.190:43451)
0924 20:10:56.373569 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.373573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.374446 (+ 873us) server_negotiation.cc:407] Connection header received
0924 20:10:56.375900 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.375903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.375989 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.376072 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:56.377340 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.378186 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.380026 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.380242 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.380321 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.383125 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:56.383129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:56.383131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:56.383154 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:56.384571 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.385153 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.385156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.385158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.385240 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.385579 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.385584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.385585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.385838 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:56.385945 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.386431 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.386536 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:10:56.387678 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.373835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:37937)
0924 20:10:56.374304 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.374346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.375650 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.376249 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.376256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.376704 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:56.377196 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.377209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.378335 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.378340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.379850 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.379865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.380393 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.380400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.380632 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.381287 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:56.381303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:56.382962 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:56.384719 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:56.384725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:56.384742 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:56.385055 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:56.385334 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:56.385337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:56.385339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:56.385487 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:56.385976 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:56.385982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:56.386219 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.386384 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.386644 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260924 20:10:56.388893 16193 heartbeater.cc:347] Connected to a master server at 127.0.115.190:43451
I20260924 20:10:56.389148 16193 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:56.389635 16193 heartbeater.cc:511] Master 127.0.115.190:43451 requested a full tablet report, sending...
I20260924 20:10:56.390785 15868 ts_manager.cc:194] Registered new tserver with Master: 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
I20260924 20:10:56.391644 15868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:37937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:10:56.404855 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:43451
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:10:56.516273 16198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:10:56.516525 16198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:10:56.516568 16198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:10:56.520505 16198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:10:56.520555 16198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:10:56.520612 16198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:10:56.525820 16198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:10:56.526795 16198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:10:56.527732 16198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:10:56.537286 16203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:56.537371 16198 server_base.cc:1034] running on GCE node
W20260924 20:10:56.537295 16206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:10:56.537295 16204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:10:56.537882 16198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:10:56.538481 16198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:10:56.539645 16198 hybrid_clock.cc:648] HybridClock initialized: now 1790280656539627 us; error 40 us; skew 500 ppm
Sep 24 20:10:56 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:10:56.542543 16198 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:10:56.543406 16198 webserver.cc:460] Webserver started at http://127.0.115.131:40023/ using document root <none> and password file <none>
I20260924 20:10:56.544035 16198 fs_manager.cc:359] Metadata directory not provided
I20260924 20:10:56.544122 16198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:10:56.544359 16198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:10:56.546393 16198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "123fb46fff0447a291547696c9dd9d9e"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "9ee2f550446864111344560488d561b3"
server_key_iv: "02f4e05f2c4964b7919e90251e613e32"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.546957 16198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "123fb46fff0447a291547696c9dd9d9e"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "9ee2f550446864111344560488d561b3"
server_key_iv: "02f4e05f2c4964b7919e90251e613e32"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.550589 16198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 20:10:56.553095 16213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.554306 16198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:10:56.554433 16198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "123fb46fff0447a291547696c9dd9d9e"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "9ee2f550446864111344560488d561b3"
server_key_iv: "02f4e05f2c4964b7919e90251e613e32"
server_key_version: "encryptionkey@0"
I20260924 20:10:56.554540 16198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:10:56.570148 16198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:10:56.573233 16198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:10:56.573441 16198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:10:56.574054 16198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:10:56.574862 16198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:10:56.574928 16198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.574995 16198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:10:56.575042 16198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:10:56.583600 16198 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:46561
I20260924 20:10:56.583662 16326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:46561 every 8 connection(s)
I20260924 20:10:56.584549 16198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:10:56.591636 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16198
I20260924 20:10:56.591744 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:10:56.592099 462 external_mini_cluster.cc:1428] Setting key b4c8df7a6e424e3b396e7c2ea2ff4b99
Sep 24 20:10:56 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:10:56.597883 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.586396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:38895 (local address 127.0.115.190:43451)
0924 20:10:56.586543 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.586548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.587626 (+ 1078us) server_negotiation.cc:407] Connection header received
0924 20:10:56.588509 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.588512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.588558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.588632 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:56.589763 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.590306 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.591462 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.591722 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.591838 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.594725 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:56.594728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:56.594730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:56.594758 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:56.596103 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.596570 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.596572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.596574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.596646 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.596921 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.596924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.596925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.597094 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:56.597169 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.597613 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.597747 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:10:56.598626 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.586842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:38895)
0924 20:10:56.587476 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.587517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.588330 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.588793 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.588799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.589158 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:10:56.589629 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.589642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.590409 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.590411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.591318 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.591327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.591909 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.591915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.592159 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.592912 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:10:56.592951 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:10:56.594587 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:10:56.596210 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:56.596215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:56.596228 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:56.596490 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:56.596718 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:10:56.596720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:10:56.596721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:10:56.596848 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:10:56.597175 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:10:56.597180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:10:56.597410 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.597562 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.597809 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":109,"threads_started":1}
I20260924 20:10:56.599694 16327 heartbeater.cc:347] Connected to a master server at 127.0.115.190:43451
I20260924 20:10:56.599915 16327 heartbeater.cc:464] Registering TS with master...
I20260924 20:10:56.600396 16327 heartbeater.cc:511] Master 127.0.115.190:43451 requested a full tablet report, sending...
I20260924 20:10:56.601361 15868 ts_manager.cc:194] Registered new tserver with Master: 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561)
I20260924 20:10:56.601974 15868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:38895
I20260924 20:10:56.606011 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:10:56.614778 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.608025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.0.115.190:43451)
0924 20:10:56.608163 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.608168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.608371 (+ 203us) server_negotiation.cc:407] Connection header received
0924 20:10:56.608468 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.608470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.608509 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.608573 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:10:56.609203 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.609697 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.610487 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.610689 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.610764 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.612121 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:10:56.612124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:10:56.612126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:10:56.612145 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:10:56.613330 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.613707 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.613709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.613710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.613781 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:10:56.614012 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:10:56.614014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:10:56.614015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:10:56.614216 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:10:56.614327 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.614494 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.614590 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 20:10:56.617641 15868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:10:56.620832 15868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:10:56.636229 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.630852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35647 (local address 127.0.0.1:43774)
0924 20:10:56.631613 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.631624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.631746 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.632212 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.632215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.632232 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:56.632343 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.632349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.633315 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.633320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.634638 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.634648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.635613 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.635617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.635733 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.635797 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.635908 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.636063 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":22,"thread_start_us":85,"threads_started":1}
I20260924 20:10:56.636250 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.631090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.0.115.130:35647)
0924 20:10:56.631439 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.631451 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.631757 (+ 306us) server_negotiation.cc:407] Connection header received
0924 20:10:56.631864 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.631869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.632089 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.632232 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:56.632438 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.633185 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.634757 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.635482 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.635602 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.635765 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.635993 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.636095 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":162,"threads_started":1}
I20260924 20:10:56.636754 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.629867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:40017 (local address 127.0.0.1:45162)
0924 20:10:56.631125 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.631138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.631220 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.631853 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.631856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.631871 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:56.632012 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.632019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.633610 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.633613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.634961 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.634972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.636261 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.636267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.636439 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.636467 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.636552 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.636624 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":55,"thread_start_us":117,"threads_started":1}
I20260924 20:10:56.636833 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.629864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.115.131:46561)
0924 20:10:56.630986 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.630999 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.631025 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:10:56.631107 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.631112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.631336 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.631521 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:56.632290 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.633438 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.635130 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.635750 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.635851 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.636023 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.636576 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.636681 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":142,"threads_started":1}
I20260924 20:10:56.636493 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.629729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:46561 (local address 127.0.0.1:49226)
0924 20:10:56.630586 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.630640 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.630821 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.631737 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.631797 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.631824 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:56.632026 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.632034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.633610 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.633613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.634961 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.634972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.636083 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.636086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.636179 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.636200 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.636283 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.636365 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":177,"thread_start_us":84,"threads_started":1}
I20260924 20:10:56.637126 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.630024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.0.115.129:40017)
0924 20:10:56.630632 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.630648 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.631316 (+ 668us) server_negotiation.cc:407] Connection header received
0924 20:10:56.631398 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.631405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.631591 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.631749 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:56.632215 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.633438 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.635095 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.635925 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.636044 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.636182 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.636598 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.636703 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":97,"threads_started":1}
I20260924 20:10:56.639621 16261 tablet_service.cc:1467] Processing CreateTablet for tablet c0f8056b1acd409193f1b3a93dc93992 (DEFAULT_TABLE table=test-table [id=a4728a6f59c14ae089e506cbf87953bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:56.639618 16127 tablet_service.cc:1467] Processing CreateTablet for tablet c0f8056b1acd409193f1b3a93dc93992 (DEFAULT_TABLE table=test-table [id=a4728a6f59c14ae089e506cbf87953bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:56.640612 16127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f8056b1acd409193f1b3a93dc93992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:56.640622 16261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f8056b1acd409193f1b3a93dc93992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:56.644151 15993 tablet_service.cc:1467] Processing CreateTablet for tablet c0f8056b1acd409193f1b3a93dc93992 (DEFAULT_TABLE table=test-table [id=a4728a6f59c14ae089e506cbf87953bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:10:56.645248 15993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f8056b1acd409193f1b3a93dc93992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:10:56.647166 16344 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Bootstrap starting.
I20260924 20:10:56.648450 16345 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Bootstrap starting.
I20260924 20:10:56.649655 16344 tablet_bootstrap.cc:654] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:56.650386 16344 log.cc:824] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:56.650957 16345 tablet_bootstrap.cc:654] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:56.651831 16345 log.cc:824] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:56.652484 16344 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: No bootstrap required, opened a new log
I20260924 20:10:56.652714 16344 ts_tablet_manager.cc:1397] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:10:56.653237 16347 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap starting.
I20260924 20:10:56.655082 16347 tablet_bootstrap.cc:654] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Neither blocks nor log segments found. Creating new log.
I20260924 20:10:56.655736 16344 raft_consensus.cc:348] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.656003 16344 raft_consensus.cc:374] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:56.656083 16344 raft_consensus.cc:729] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123fb46fff0447a291547696c9dd9d9e, State: Initialized, Role: FOLLOWER
I20260924 20:10:56.656136 16347 log.cc:824] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:10:56.656603 16344 consensus_queue.cc:260] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.657356 16327 heartbeater.cc:503] Master 127.0.115.190:43451 was elected leader, sending a full tablet report...
I20260924 20:10:56.657653 16345 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: No bootstrap required, opened a new log
I20260924 20:10:56.657866 16344 ts_tablet_manager.cc:1428] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 20:10:56.657894 16345 ts_tablet_manager.cc:1397] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 20:10:56.658115 16347 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: No bootstrap required, opened a new log
I20260924 20:10:56.658346 16347 ts_tablet_manager.cc:1397] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:10:56.661859 16345 raft_consensus.cc:348] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.662144 16345 raft_consensus.cc:374] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:56.662220 16345 raft_consensus.cc:729] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b025e00fa64475f8a2c69cbc8bc357f, State: Initialized, Role: FOLLOWER
I20260924 20:10:56.662122 16347 raft_consensus.cc:348] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.662356 16347 raft_consensus.cc:374] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:10:56.662407 16347 raft_consensus.cc:729] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcf6925e1b4e99ae523c7de879803c, State: Initialized, Role: FOLLOWER
I20260924 20:10:56.662712 16345 consensus_queue.cc:260] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.662762 16347 consensus_queue.cc:260] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.663345 16059 heartbeater.cc:503] Master 127.0.115.190:43451 was elected leader, sending a full tablet report...
I20260924 20:10:56.663719 16345 ts_tablet_manager.cc:1428] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:10:56.664106 16193 heartbeater.cc:503] Master 127.0.115.190:43451 was elected leader, sending a full tablet report...
I20260924 20:10:56.664707 16347 ts_tablet_manager.cc:1428] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:10:56.785056 16350 raft_consensus.cc:482] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:10:56.785269 16350 raft_consensus.cc:504] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.786439 16350 leader_election.cc:288] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017), 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
I20260924 20:10:56.790023 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.786711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:40017 (local address 127.0.115.131:49629)
0924 20:10:56.786843 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.786856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.786974 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.787220 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.787222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.787239 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:56.787367 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.787373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.788176 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.788180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.788935 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.788943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.789597 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.789600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.789712 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.789731 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.789814 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.789887 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 20:10:56.790023 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.786839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:49629 (local address 127.0.115.129:40017)
0924 20:10:56.786982 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.786987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.787003 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:10:56.787062 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.787068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.787120 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.787207 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:56.787484 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.788067 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.789064 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.789484 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.789609 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.789667 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.789834 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.789913 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:10:56.790555 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.787158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:41657 (local address 127.0.115.130:35647)
0924 20:10:56.787308 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.787313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.787505 (+ 192us) server_negotiation.cc:407] Connection header received
0924 20:10:56.787628 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.787635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.787677 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.787754 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:10:56.788047 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.788545 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.789576 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.789955 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.790088 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.790136 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.790349 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.790440 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:10:56.790786 16013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f8056b1acd409193f1b3a93dc93992" candidate_uuid: "123fb46fff0447a291547696c9dd9d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcf6925e1b4e99ae523c7de879803c" is_pre_election: true
I20260924 20:10:56.790902 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.787024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35647 (local address 127.0.115.131:41657)
0924 20:10:56.787404 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.787424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:10:56.787538 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:10:56.787772 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:10:56.787775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:10:56.787798 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:10:56.787915 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.787920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.788653 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.788658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.789450 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:10:56.789458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.790088 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.790092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:10:56.790190 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.790220 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:10:56.790699 (+ 479us) client_negotiation.cc:240] Negotiation successful
0924 20:10:56.790770 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":117,"threads_started":1}
I20260924 20:10:56.791134 16013 raft_consensus.cc:2454] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123fb46fff0447a291547696c9dd9d9e in term 0.
I20260924 20:10:56.791530 16147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f8056b1acd409193f1b3a93dc93992" candidate_uuid: "123fb46fff0447a291547696c9dd9d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" is_pre_election: true
I20260924 20:10:56.791632 16214 leader_election.cc:302] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123fb46fff0447a291547696c9dd9d9e, 84bcf6925e1b4e99ae523c7de879803c; no voters:
I20260924 20:10:56.791818 16147 raft_consensus.cc:2454] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123fb46fff0447a291547696c9dd9d9e in term 0.
I20260924 20:10:56.791957 16350 raft_consensus.cc:2781] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:10:56.792040 16350 raft_consensus.cc:482] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:10:56.792105 16350 raft_consensus.cc:3037] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:56.793298 16350 raft_consensus.cc:504] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.793670 16350 leader_election.cc:288] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [CANDIDATE]: Term 1 election: Requested vote from peers 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017), 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
I20260924 20:10:56.794001 16013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f8056b1acd409193f1b3a93dc93992" candidate_uuid: "123fb46fff0447a291547696c9dd9d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcf6925e1b4e99ae523c7de879803c"
I20260924 20:10:56.794047 16147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f8056b1acd409193f1b3a93dc93992" candidate_uuid: "123fb46fff0447a291547696c9dd9d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b025e00fa64475f8a2c69cbc8bc357f"
I20260924 20:10:56.794098 16013 raft_consensus.cc:3037] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:56.794180 16147 raft_consensus.cc:3037] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:10:56.795239 16013 raft_consensus.cc:2454] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123fb46fff0447a291547696c9dd9d9e in term 1.
I20260924 20:10:56.795246 16147 raft_consensus.cc:2454] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123fb46fff0447a291547696c9dd9d9e in term 1.
I20260924 20:10:56.795524 16214 leader_election.cc:302] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123fb46fff0447a291547696c9dd9d9e, 84bcf6925e1b4e99ae523c7de879803c; no voters:
I20260924 20:10:56.795737 16350 raft_consensus.cc:2781] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:10:56.795993 16350 raft_consensus.cc:686] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 1 LEADER]: Becoming Leader. State: Replica: 123fb46fff0447a291547696c9dd9d9e, State: Running, Role: LEADER
I20260924 20:10:56.796331 16350 consensus_queue.cc:237] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:10:56.799396 15867 catalog_manager.cc:5495] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 123fb46fff0447a291547696c9dd9d9e (127.0.115.131). New cstate: current_term: 1 leader_uuid: "123fb46fff0447a291547696c9dd9d9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } health_report { overall_health: UNKNOWN } } }
I20260924 20:10:56.825706 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.822450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.0.115.131:46561)
0924 20:10:56.822577 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:10:56.822580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:10:56.822592 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:10:56.822666 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:10:56.822668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:10:56.822721 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:10:56.822807 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:10:56.823026 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.823518 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.824334 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:10:56.824536 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:10:56.824605 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:10:56.824877 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:10:56.825307 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:10:56.825435 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 20:10:56.825545 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 20:10:56.825596 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:10:56.831902 16013 raft_consensus.cc:1261] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Refusing update from remote peer 123fb46fff0447a291547696c9dd9d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:56.831985 16147 raft_consensus.cc:1261] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 FOLLOWER]: Refusing update from remote peer 123fb46fff0447a291547696c9dd9d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:10:56.832645 16354 consensus_queue.cc:1035] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:10:56.832855 16350 consensus_queue.cc:1035] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:10:56.835683 16328 tablet.cc:2369] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:10:56.847491 16359 mvcc.cc:205] Tried to move back new op lower bound from 7332989570375815168 to 7332989570240421888. Current Snapshot: MvccSnapshot[applied={T|T < 7332989570375815168}]
I20260924 20:10:56.848268 15868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:56.848439 15868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:10:56.850838 15868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Sending DeleteTablet for 3 replicas of tablet c0f8056b1acd409193f1b3a93dc93992
I20260924 20:10:56.851699 15993 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f8056b1acd409193f1b3a93dc93992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:56 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:45162
I20260924 20:10:56.851769 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 15930
I20260924 20:10:56.851913 16127 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f8056b1acd409193f1b3a93dc93992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:56 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:43774
I20260924 20:10:56.852407 16261 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f8056b1acd409193f1b3a93dc93992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:56 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:49226
I20260924 20:10:56.852862 16367 tablet_replica.cc:330] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: stopping tablet replica
I20260924 20:10:56.853223 16367 raft_consensus.cc:2229] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:10:56.853631 16367 raft_consensus.cc:2258] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:56.855190 16367 ts_tablet_manager.cc:1905] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:56.856148 16366 tablet_replica.cc:330] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: stopping tablet replica
I20260924 20:10:56.856487 16366 raft_consensus.cc:2229] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:10:56.856828 16366 raft_consensus.cc:2258] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:10:56.858199 16367 ts_tablet_manager.cc:1918] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:56.858341 16367 log.cc:1196] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/c0f8056b1acd409193f1b3a93dc93992
I20260924 20:10:56.858362 16366 ts_tablet_manager.cc:1905] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:10:56.858721 16367 ts_tablet_manager.cc:1939] T c0f8056b1acd409193f1b3a93dc93992 P 123fb46fff0447a291547696c9dd9d9e: Deleting consensus metadata
I20260924 20:10:56.860131 15853 catalog_manager.cc:4841] TS 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561): tablet c0f8056b1acd409193f1b3a93dc93992 (table test-table [id=a4728a6f59c14ae089e506cbf87953bb]) successfully deleted
I20260924 20:10:56.861204 16366 ts_tablet_manager.cc:1918] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:10:56.861299 16366 log.cc:1196] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/c0f8056b1acd409193f1b3a93dc93992
I20260924 20:10:56.861588 16366 ts_tablet_manager.cc:1939] T c0f8056b1acd409193f1b3a93dc93992 P 9b025e00fa64475f8a2c69cbc8bc357f: Deleting consensus metadata
I20260924 20:10:56.862843 15852 catalog_manager.cc:4841] TS 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647): tablet c0f8056b1acd409193f1b3a93dc93992 (table test-table [id=a4728a6f59c14ae089e506cbf87953bb]) successfully deleted
I20260924 20:10:56.863811 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 16064
W20260924 20:10:56.863831 15852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:40017 (error 108)
I20260924 20:10:56.865113 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:10:56.864783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:40017 (local address 127.0.0.1:45176)
0924 20:10:56.864971 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 20:10:56.865035 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260924 20:10:56.865257 15852 catalog_manager.cc:4584] TS 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0f8056b1acd409193f1b3a93dc93992: Network error: Client connection negotiation failed: client connection to 127.0.115.129:40017: connect: Connection refused (error 111)
I20260924 20:10:56.870383 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 16198
I20260924 20:10:56.876444 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 15836
I20260924 20:11:16.883376 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:43451
--webserver_interface=127.0.115.190
--webserver_port=34743
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:16.995716 16369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:16.996075 16369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:16.996176 16369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:17.000131 16369 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:11:17.000178 16369 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:11:17.000193 16369 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:11:17.000207 16369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:11:17.000257 16369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:17.000293 16369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:17.000324 16369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:11:17.000356 16369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:11:17.005565 16369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:43451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:17.006672 16369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:17.007764 16369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:17.020767 16376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:17.020733 16375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:17.020749 16369 server_base.cc:1034] running on GCE node
W20260924 20:11:17.020742 16378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:17.021546 16369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:17.022653 16369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:17.023874 16369 hybrid_clock.cc:648] HybridClock initialized: now 1790280677023829 us; error 65 us; skew 500 ppm
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:17.026721 16369 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:11:17.027648 16369 webserver.cc:460] Webserver started at http://127.0.115.190:34743/ using document root <none> and password file <none>
I20260924 20:11:17.028304 16369 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:17.028409 16369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:17.032356 16369 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:11:17.034644 16385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.035840 16369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:11:17.036054 16369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "ba9770723fb049ce916f20faec01529f"
format_stamp: "Formatted at 2026-09-24 20:10:55 on dist-test-slave-2pm7"
server_key: "6cacbc369e2140fc9007f1cf5225b920"
server_key_iv: "f38ef47ce39aa64b740be75e7ff2b714"
server_key_version: "encryptionkey@0"
I20260924 20:11:17.036659 16369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:17.051539 16369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:17.054721 16369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:17.054935 16369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:17.062621 16369 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:43451
I20260924 20:11:17.062947 16437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:43451 every 8 connection(s)
I20260924 20:11:17.063807 16369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:11:17.068385 16438 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:11:17.069988 16438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Bootstrap starting.
I20260924 20:11:17.070588 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16369
I20260924 20:11:17.071085 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:40017
--local_ip_for_outbound_sockets=127.0.115.129
--tserver_master_addrs=127.0.115.190:43451
--webserver_port=38063
--webserver_interface=127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:11:17.088228 16438 log.cc:824] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:17.097234 16438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:11:17.097621 16438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Bootstrap complete.
I20260924 20:11:17.101701 16438 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:17.102125 16438 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9770723fb049ce916f20faec01529f, State: Initialized, Role: FOLLOWER
I20260924 20:11:17.102638 16438 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:17.102753 16438 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:11:17.102849 16438 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:11:17.102964 16438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:11:17.104952 16438 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:17.105290 16438 leader_election.cc:302] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba9770723fb049ce916f20faec01529f; no voters:
I20260924 20:11:17.105650 16438 leader_election.cc:288] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:11:17.105734 16442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:11:17.105988 16442 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 LEADER]: Becoming Leader. State: Replica: ba9770723fb049ce916f20faec01529f, State: Running, Role: LEADER
I20260924 20:11:17.106354 16442 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:17.106995 16438 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:11:17.107787 16443 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba9770723fb049ce916f20faec01529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } } }
I20260924 20:11:17.107913 16443 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:17.108587 16450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:11:17.108543 16444 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9770723fb049ce916f20faec01529f. Latest consensus state: current_term: 2 leader_uuid: "ba9770723fb049ce916f20faec01529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } } }
I20260924 20:11:17.109251 16444 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:17.113446 16450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:11:17.114178 16450 catalog_manager.cc:1259] Loaded cluster ID: ef8648ae908e4845a1fbd3eeb04aa781
I20260924 20:11:17.114238 16450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:11:17.116904 16450 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:11:17.117659 16450 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Loaded TSK: 0
I20260924 20:11:17.139679 16450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Generated new TSK 1
I20260924 20:11:17.140038 16450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:11:17.193339 16440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:17.193619 16440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:17.193665 16440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:17.197916 16440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:17.197983 16440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:17.198050 16440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:11:17.203904 16440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:40017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=38063
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:17.205324 16440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:17.206501 16440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:17.217147 16465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:17.217243 16466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:17.217192 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:17.217602 16440 server_base.cc:1034] running on GCE node
I20260924 20:11:17.218086 16440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:17.218932 16440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:17.220130 16440 hybrid_clock.cc:648] HybridClock initialized: now 1790280677220092 us; error 46 us; skew 500 ppm
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:17.226876 16440 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:11:17.228004 16440 webserver.cc:460] Webserver started at http://127.0.115.129:38063/ using document root <none> and password file <none>
I20260924 20:11:17.228607 16440 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:17.228682 16440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:17.233057 16440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:11:17.235390 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.236842 16440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:11:17.237010 16440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "84bcf6925e1b4e99ae523c7de879803c"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "116d7c88e3859e61ff837ab3b6e049e2"
server_key_iv: "075c8610c300f61be36ec91ec00781d9"
server_key_version: "encryptionkey@0"
I20260924 20:11:17.237609 16440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:17.268251 16440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:17.271795 16440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:17.272040 16440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:17.272701 16440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:17.273819 16482 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:11:17.278184 16440 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:11:17.278246 16440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 20:11:17.278322 16440 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:11:17.280122 16440 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:11:17.280169 16440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260924 20:11:17.280471 16482 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap starting.
I20260924 20:11:17.290701 16440 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:40017
I20260924 20:11:17.290786 16589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:40017 every 8 connection(s)
I20260924 20:11:17.291704 16440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:11:17.298923 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16440
I20260924 20:11:17.299394 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:35647
--local_ip_for_outbound_sockets=127.0.115.130
--tserver_master_addrs=127.0.115.190:43451
--webserver_port=39655
--webserver_interface=127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:11:17.300964 16482 log.cc:824] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:17.310456 16482 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:11:17.310981 16482 tablet_bootstrap.cc:492] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap complete.
I20260924 20:11:17.312386 16482 ts_tablet_manager.cc:1397] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.007s
I20260924 20:11:17.313376 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.294324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:58567 (local address 127.0.115.190:43451)
0924 20:11:17.294739 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.294758 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.295079 (+ 321us) server_negotiation.cc:407] Connection header received
0924 20:11:17.296265 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.296281 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.296715 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.297186 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:17.298271 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.299915 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.301653 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.302473 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.302661 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.306132 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:17.306140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:17.306166 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:17.306201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:17.308621 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.309420 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.309427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.309432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.309592 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.310064 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.310067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.310069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.310581 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:17.310730 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.312149 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.312450 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260924 20:11:17.314465 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.294194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:58567)
0924 20:11:17.294896 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.294949 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.295782 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.297215 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.297222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.297601 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:17.298062 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.298077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.300111 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.300118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.301506 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.301520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.302637 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.302642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.302831 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.303558 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:17.303580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:17.305702 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:17.308769 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.308778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.308802 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.309244 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.309722 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.309728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.309731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.309941 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.310809 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:17.310819 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:17.311145 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.311361 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.311740 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":152,"threads_started":1}
I20260924 20:11:17.315485 16482 raft_consensus.cc:348] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:11:17.316001 16482 raft_consensus.cc:729] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcf6925e1b4e99ae523c7de879803c, State: Initialized, Role: FOLLOWER
I20260924 20:11:17.316519 16482 consensus_queue.cc:260] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } }
I20260924 20:11:17.317297 16590 heartbeater.cc:347] Connected to a master server at 127.0.115.190:43451
I20260924 20:11:17.317533 16590 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:17.317646 16482 ts_tablet_manager.cc:1428] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260924 20:11:17.318084 16590 heartbeater.cc:511] Master 127.0.115.190:43451 requested a full tablet report, sending...
I20260924 20:11:17.320797 16402 ts_manager.cc:194] Registered new tserver with Master: 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017)
I20260924 20:11:17.324134 16402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:58567
I20260924 20:11:17.327000 16590 heartbeater.cc:503] Master 127.0.115.190:43451 was elected leader, sending a full tablet report...
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.129@KRBTEST.COM
I20260924 20:11:17.334317 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.323670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:40017 (local address 127.0.0.1:45460)
0924 20:11:17.324126 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.324172 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.324302 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.325338 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.325345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.325709 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:17.325815 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.325822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.326814 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.326818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.327597 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.327608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.328254 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.328258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.328381 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.328994 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:17.329006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:17.330509 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:17.332485 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.332494 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.332499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.332729 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.333085 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.333088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.333089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.333165 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.333626 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:17.333632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:17.333755 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.333911 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.334131 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":125,"threads_started":1}
I20260924 20:11:17.334317 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.324545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.0.115.129:40017)
0924 20:11:17.324845 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.324858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.324883 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:11:17.324985 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.324990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.325195 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.325335 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:17.325944 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.326693 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.327713 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.328136 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.328251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.330646 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:17.330656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:17.330670 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:17.330701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:17.332358 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.332854 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.332859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.332860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.332935 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.333260 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.333264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.333266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.333512 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:17.333652 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.333991 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.334148 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260924 20:11:17.335410 16524 tablet_service.cc:1514] Processing DeleteTablet for tablet c0f8056b1acd409193f1b3a93dc93992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:10:56 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:45460
I20260924 20:11:17.336326 16599 tablet_replica.cc:330] stopping tablet replica
I20260924 20:11:17.336719 16599 raft_consensus.cc:2229] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:11:17.337003 16599 raft_consensus.cc:2258] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:17.338213 16599 ts_tablet_manager.cc:1905] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:17.343222 16599 ts_tablet_manager.cc:1918] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:17.343340 16599 log.cc:1196] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal/wals/c0f8056b1acd409193f1b3a93dc93992
I20260924 20:11:17.343640 16599 ts_tablet_manager.cc:1939] T c0f8056b1acd409193f1b3a93dc93992 P 84bcf6925e1b4e99ae523c7de879803c: Deleting consensus metadata
I20260924 20:11:17.346092 16386 catalog_manager.cc:4841] TS 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017): tablet c0f8056b1acd409193f1b3a93dc93992 (table test-table [id=a4728a6f59c14ae089e506cbf87953bb]) successfully deleted
W20260924 20:11:17.427595 16594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:17.427872 16594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:17.427949 16594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:17.432191 16594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:17.432260 16594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:17.432360 16594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:11:17.437855 16594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:35647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=39655
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:17.438997 16594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:17.440124 16594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:17.450201 16604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:17.450201 16605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:17.450302 16594 server_base.cc:1034] running on GCE node
W20260924 20:11:17.450228 16607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:17.450896 16594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:17.451591 16594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:17.452793 16594 hybrid_clock.cc:648] HybridClock initialized: now 1790280677452755 us; error 49 us; skew 500 ppm
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:17.455684 16594 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:11:17.456784 16594 webserver.cc:460] Webserver started at http://127.0.115.130:39655/ using document root <none> and password file <none>
I20260924 20:11:17.457365 16594 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:17.457480 16594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:17.461654 16594 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:11:17.463868 16614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.465271 16594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:11:17.465426 16594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "9b025e00fa64475f8a2c69cbc8bc357f"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "85c76b031d1c6a2115030ac7d5542034"
server_key_iv: "e013cf0c319ca9118bbdfc331245a0c3"
server_key_version: "encryptionkey@0"
I20260924 20:11:17.466014 16594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:17.490377 16594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:17.494320 16594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:17.494544 16594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:17.495193 16594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:17.496155 16594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:17.496222 16594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.496290 16594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:17.496337 16594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.506006 16594 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:35647
I20260924 20:11:17.506088 16727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:35647 every 8 connection(s)
I20260924 20:11:17.507227 16594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:11:17.507730 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16594
I20260924 20:11:17.508301 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:46561
--local_ip_for_outbound_sockets=127.0.115.131
--tserver_master_addrs=127.0.115.190:43451
--webserver_port=40023
--webserver_interface=127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:17.523267 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.509422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:34349 (local address 127.0.115.190:43451)
0924 20:11:17.509796 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.509804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.511104 (+ 1300us) server_negotiation.cc:407] Connection header received
0924 20:11:17.512101 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.512107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.512181 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.512284 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:17.513577 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.514542 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.515788 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.516267 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.516403 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.519230 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:17.519234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:17.519239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:17.519275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:17.521103 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.521619 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.521623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.521625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.521714 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.522025 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.522028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.522030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.522251 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:17.522346 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.522940 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.523081 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260924 20:11:17.524194 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.510012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:34349)
0924 20:11:17.510906 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.510968 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.511832 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.512457 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.512467 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.512872 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:17.513399 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.513415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.514700 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.514703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.515646 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.515657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.516429 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.516432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.516596 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.517267 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:17.517284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:17.519038 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:17.521223 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.521228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.521241 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.521521 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.521801 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.521803 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.521805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.521940 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.522349 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:17.522355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:17.522681 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.522864 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.523143 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":133,"threads_started":1}
I20260924 20:11:17.525449 16728 heartbeater.cc:347] Connected to a master server at 127.0.115.190:43451
I20260924 20:11:17.525719 16728 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:17.526358 16728 heartbeater.cc:511] Master 127.0.115.190:43451 requested a full tablet report, sending...
I20260924 20:11:17.527606 16402 ts_manager.cc:194] Registered new tserver with Master: 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
I20260924 20:11:17.528278 16402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:34349
W20260924 20:11:17.626673 16730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:17.626940 16730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:17.626983 16730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:17.631191 16730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:17.631259 16730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:17.631350 16730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:11:17.636790 16730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:46561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=40023
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:17.637914 16730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:17.639056 16730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:17.649310 16737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:17.649291 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:17.649286 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:17.649567 16730 server_base.cc:1034] running on GCE node
I20260924 20:11:17.650044 16730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:17.650717 16730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:17.651918 16730 hybrid_clock.cc:648] HybridClock initialized: now 1790280677651890 us; error 49 us; skew 500 ppm
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:17.654841 16730 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:11:17.655815 16730 webserver.cc:460] Webserver started at http://127.0.115.131:40023/ using document root <none> and password file <none>
I20260924 20:11:17.656528 16730 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:17.656649 16730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:17.661082 16730 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:11:17.663353 16746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.664707 16730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:11:17.664857 16730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "123fb46fff0447a291547696c9dd9d9e"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "9ee2f550446864111344560488d561b3"
server_key_iv: "02f4e05f2c4964b7919e90251e613e32"
server_key_version: "encryptionkey@0"
I20260924 20:11:17.665431 16730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:17.697953 16730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:17.701503 16730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:17.701715 16730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:17.702364 16730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:17.703297 16730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:17.703370 16730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.703444 16730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:17.703495 16730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:17.713474 16730 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:46561
I20260924 20:11:17.713517 16859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:46561 every 8 connection(s)
I20260924 20:11:17.714540 16730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:11:17.716220 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16730
Sep 24 20:11:17 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:17.726054 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.716939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.0.115.190:43451)
0924 20:11:17.717159 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.717163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.717445 (+ 282us) server_negotiation.cc:407] Connection header received
0924 20:11:17.717597 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.717600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.717648 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.717780 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:17.718654 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.719528 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.720473 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.720822 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.720918 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.722326 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:17.722330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:17.722332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:17.722359 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:17.724280 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.724715 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.724718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.724720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.724815 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.725094 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.725100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.725101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.725328 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:17.725551 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.725728 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.725857 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
I20260924 20:11:17.729851 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.716825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:48239 (local address 127.0.115.190:43451)
0924 20:11:17.717006 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.717011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.718304 (+ 1293us) server_negotiation.cc:407] Connection header received
0924 20:11:17.719409 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.719411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.719450 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.719507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:17.720825 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.721552 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.722958 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.723173 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.723304 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.726237 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:17.726241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:17.726243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:17.726266 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:17.727692 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.728343 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.728348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.728350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.728434 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.728798 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.728801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.728802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.729034 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:17.729119 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.729625 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.729735 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 20:11:17.730651 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.717427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:48239)
0924 20:11:17.718175 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.718214 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.719188 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.719694 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.719701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.720104 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:17.720679 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.720698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.721689 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.721692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.722830 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.722846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.723362 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.723369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.723598 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.724311 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:17.724337 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:17.726046 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:17.727803 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.727810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.727826 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.728237 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.728534 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:17.728536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:17.728537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:17.728673 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:17.729133 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:17.729138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:17.729371 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.729538 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.729797 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":123,"threads_started":1}
I20260924 20:11:17.731806 16860 heartbeater.cc:347] Connected to a master server at 127.0.115.190:43451
I20260924 20:11:17.732203 16860 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:17.732792 16860 heartbeater.cc:511] Master 127.0.115.190:43451 requested a full tablet report, sending...
I20260924 20:11:17.734089 16402 ts_manager.cc:194] Registered new tserver with Master: 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561)
I20260924 20:11:17.734701 16402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:48239
I20260924 20:11:17.740237 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:11:17.744657 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.740866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.0.115.190:43451)
0924 20:11:17.741049 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.741054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.741207 (+ 153us) server_negotiation.cc:407] Connection header received
0924 20:11:17.741428 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.741431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.741540 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.741631 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:11:17.741871 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.742398 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.743376 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.743605 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.743696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.744143 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:11:17.744285 (+ 142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 20:11:17.744528 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33100: token expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 20:11:17.744732 16592 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33100: token expired
W20260924 20:11:17.745270 16331 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.115.190:43451)
I20260924 20:11:17.752982 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.745613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.0.115.190:43451)
0924 20:11:17.745754 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.745758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.745771 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:11:17.745883 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.745886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.745989 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.746077 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:17.746699 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.747193 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.748054 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.748273 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.748384 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.749770 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:17.749774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:17.749776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:17.749801 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:17.751281 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.751709 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.751713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.751714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.751802 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:17.752120 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:17.752123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:17.752125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:17.752299 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:17.752452 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.752694 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.752809 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:11:17.755599 16402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:11:17.759068 16402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:11:17.767859 16524 tablet_service.cc:1467] Processing CreateTablet for tablet 1573a3418de44b4ab5c8415917800d31 (DEFAULT_TABLE table=test-table [id=72ee17368d874e1b8272c3052dbe14d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:17.768630 16524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1573a3418de44b4ab5c8415917800d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:17.771613 16482 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap starting.
I20260924 20:11:17.772573 16482 tablet_bootstrap.cc:654] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:17.772784 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.766382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:46561 (local address 127.0.0.1:51554)
0924 20:11:17.766706 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.766722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.766819 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.767641 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.767646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.767670 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:17.767787 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.767793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.769564 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.769567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.771038 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.771049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.772278 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.772281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.772380 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.772438 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.772540 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.772633 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":51}
I20260924 20:11:17.772869 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.766672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.0.115.131:46561)
0924 20:11:17.767095 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.767110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.767133 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:11:17.767219 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.767224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.767455 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.767627 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:17.767966 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.769398 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.771177 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.772142 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.772270 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.772426 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.772584 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.772690 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":175,"threads_started":1}
I20260924 20:11:17.773001 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.766650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35647 (local address 127.0.0.1:47480)
0924 20:11:17.766951 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.766964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.767040 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.768343 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.768348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.768370 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:17.768470 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.768477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.770045 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.770048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.771088 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.771099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.772553 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.772557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.772659 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.772683 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.772764 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.772833 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":2,"thread_start_us":151,"threads_started":1}
I20260924 20:11:17.773515 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.767189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.0.115.130:35647)
0924 20:11:17.767575 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.767593 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.767616 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:11:17.767694 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.767700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.767965 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.768140 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:17.768627 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.769900 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.771263 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.772142 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.772270 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.772425 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.773272 (+ 847us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.773373 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":165,"threads_started":1}
I20260924 20:11:17.774578 16482 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: No bootstrap required, opened a new log
I20260924 20:11:17.774667 16482 ts_tablet_manager.cc:1397] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:11:17.775074 16482 raft_consensus.cc:348] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.775195 16482 raft_consensus.cc:374] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:17.775238 16482 raft_consensus.cc:729] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcf6925e1b4e99ae523c7de879803c, State: Initialized, Role: FOLLOWER
I20260924 20:11:17.775357 16482 consensus_queue.cc:260] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.775614 16482 ts_tablet_manager.cc:1428] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:11:17.776274 16794 tablet_service.cc:1467] Processing CreateTablet for tablet 1573a3418de44b4ab5c8415917800d31 (DEFAULT_TABLE table=test-table [id=72ee17368d874e1b8272c3052dbe14d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:17.776515 16662 tablet_service.cc:1467] Processing CreateTablet for tablet 1573a3418de44b4ab5c8415917800d31 (DEFAULT_TABLE table=test-table [id=72ee17368d874e1b8272c3052dbe14d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:17.777410 16794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1573a3418de44b4ab5c8415917800d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:17.777712 16662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1573a3418de44b4ab5c8415917800d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:17.784410 16874 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Bootstrap starting.
I20260924 20:11:17.785089 16873 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Bootstrap starting.
I20260924 20:11:17.786975 16874 tablet_bootstrap.cc:654] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:17.787160 16873 tablet_bootstrap.cc:654] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:17.787815 16874 log.cc:824] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:17.787837 16873 log.cc:824] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:17.789726 16874 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: No bootstrap required, opened a new log
I20260924 20:11:17.789799 16873 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: No bootstrap required, opened a new log
I20260924 20:11:17.790004 16873 ts_tablet_manager.cc:1397] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:11:17.790016 16874 ts_tablet_manager.cc:1397] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260924 20:11:17.792932 16591 tablet.cc:2369] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:11:17.793255 16873 raft_consensus.cc:348] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.793530 16873 raft_consensus.cc:374] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:17.793614 16873 raft_consensus.cc:729] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123fb46fff0447a291547696c9dd9d9e, State: Initialized, Role: FOLLOWER
I20260924 20:11:17.793488 16874 raft_consensus.cc:348] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.793812 16874 raft_consensus.cc:374] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:17.793920 16874 raft_consensus.cc:729] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b025e00fa64475f8a2c69cbc8bc357f, State: Initialized, Role: FOLLOWER
I20260924 20:11:17.794126 16873 consensus_queue.cc:260] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.794382 16874 consensus_queue.cc:260] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.794834 16860 heartbeater.cc:503] Master 127.0.115.190:43451 was elected leader, sending a full tablet report...
I20260924 20:11:17.795038 16728 heartbeater.cc:503] Master 127.0.115.190:43451 was elected leader, sending a full tablet report...
I20260924 20:11:17.795470 16874 ts_tablet_manager.cc:1428] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:11:17.796214 16873 ts_tablet_manager.cc:1428] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:11:17.903120 16878 raft_consensus.cc:482] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:17.903411 16878 raft_consensus.cc:504] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.904785 16878 leader_election.cc:288] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561), 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017)
I20260924 20:11:17.909600 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.905025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:46561 (local address 127.0.115.130:53383)
0924 20:11:17.905168 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.905188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.905292 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.905672 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.905676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.905695 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:17.905820 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.905825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.906773 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.906776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.908230 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.908240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.909108 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.909111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.909231 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.909262 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.909341 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.909443 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":37}
I20260924 20:11:17.909598 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.905184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:53383 (local address 127.0.115.131:46561)
0924 20:11:17.905349 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.905354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.905371 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:11:17.905435 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.905441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.905546 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.905640 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:17.905941 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.906659 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.908349 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.908991 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.909113 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.909169 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.909392 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.909476 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 20:11:17.909870 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.905338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:40977 (local address 127.0.115.129:40017)
0924 20:11:17.905677 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.905681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.905699 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:11:17.905756 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.905759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.905819 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.905980 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:17.906332 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.906989 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.908506 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.909291 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.909433 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.909581 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.909660 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.909735 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260924 20:11:17.909878 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.905183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:40017 (local address 127.0.115.130:40977)
0924 20:11:17.905524 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:17.905539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:17.905661 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:17.906020 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:17.906026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:17.906050 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:17.906194 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.906199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.907152 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.907156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.908382 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:17.908393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.909437 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.909440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:17.909541 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.909563 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:11:17.909640 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:11:17.909713 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":41,"thread_start_us":109,"threads_started":1}
I20260924 20:11:17.910478 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573a3418de44b4ab5c8415917800d31" candidate_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123fb46fff0447a291547696c9dd9d9e" is_pre_election: true
I20260924 20:11:17.910638 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573a3418de44b4ab5c8415917800d31" candidate_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcf6925e1b4e99ae523c7de879803c" is_pre_election: true
I20260924 20:11:17.910867 16814 raft_consensus.cc:2454] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b025e00fa64475f8a2c69cbc8bc357f in term 0.
I20260924 20:11:17.911000 16544 raft_consensus.cc:2454] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b025e00fa64475f8a2c69cbc8bc357f in term 0.
I20260924 20:11:17.911376 16616 leader_election.cc:302] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123fb46fff0447a291547696c9dd9d9e, 9b025e00fa64475f8a2c69cbc8bc357f; no voters:
I20260924 20:11:17.911689 16878 raft_consensus.cc:2781] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:11:17.911779 16878 raft_consensus.cc:482] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:11:17.911826 16878 raft_consensus.cc:3037] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:17.912904 16878 raft_consensus.cc:504] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.913283 16878 leader_election.cc:288] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [CANDIDATE]: Term 1 election: Requested vote from peers 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561), 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017)
I20260924 20:11:17.913672 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573a3418de44b4ab5c8415917800d31" candidate_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123fb46fff0447a291547696c9dd9d9e"
I20260924 20:11:17.913702 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573a3418de44b4ab5c8415917800d31" candidate_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bcf6925e1b4e99ae523c7de879803c"
I20260924 20:11:17.913842 16544 raft_consensus.cc:3037] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:17.913828 16814 raft_consensus.cc:3037] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:17.914904 16814 raft_consensus.cc:2454] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b025e00fa64475f8a2c69cbc8bc357f in term 1.
I20260924 20:11:17.914911 16544 raft_consensus.cc:2454] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b025e00fa64475f8a2c69cbc8bc357f in term 1.
I20260924 20:11:17.915226 16615 leader_election.cc:302] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84bcf6925e1b4e99ae523c7de879803c, 9b025e00fa64475f8a2c69cbc8bc357f; no voters:
I20260924 20:11:17.915421 16878 raft_consensus.cc:2781] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:17.915616 16878 raft_consensus.cc:686] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 LEADER]: Becoming Leader. State: Replica: 9b025e00fa64475f8a2c69cbc8bc357f, State: Running, Role: LEADER
I20260924 20:11:17.916057 16878 consensus_queue.cc:237] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:17.919432 16401 catalog_manager.cc:5495] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130). New cstate: current_term: 1 leader_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } health_report { overall_health: UNKNOWN } } }
W20260924 20:11:17.965562 16861 tablet.cc:2369] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:11:17.965646 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:17.961832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.115.130:35647)
0924 20:11:17.961978 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:17.961982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:17.962000 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:11:17.962077 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:17.962079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:17.962136 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:17.962224 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:11:17.962472 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.963038 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.963982 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:17.964222 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:17.964323 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:17.964651 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:11:17.965123 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:11:17.965239 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:17.965386 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 20:11:17.965473 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:11:17.971920 16544 raft_consensus.cc:1261] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Refusing update from remote peer 9b025e00fa64475f8a2c69cbc8bc357f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:17.971921 16814 raft_consensus.cc:1261] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 1 FOLLOWER]: Refusing update from remote peer 9b025e00fa64475f8a2c69cbc8bc357f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:17.972693 16881 consensus_queue.cc:1035] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:17.972895 16878 consensus_queue.cc:1035] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:17.977705 16884 mvcc.cc:205] Tried to move back new op lower bound from 7332989656965718016 to 7332989656746835968. Current Snapshot: MvccSnapshot[applied={T|T < 7332989656965718016}]
I20260924 20:11:17.981405 16887 mvcc.cc:205] Tried to move back new op lower bound from 7332989656965718016 to 7332989656746835968. Current Snapshot: MvccSnapshot[applied={T|T < 7332989656965718016}]
I20260924 20:11:17.988417 16398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:17.988631 16398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:17.991245 16398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Sending DeleteTablet for 3 replicas of tablet 1573a3418de44b4ab5c8415917800d31
I20260924 20:11:17.992064 16662 tablet_service.cc:1514] Processing DeleteTablet for tablet 1573a3418de44b4ab5c8415917800d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:17 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:47480
I20260924 20:11:17.992137 16524 tablet_service.cc:1514] Processing DeleteTablet for tablet 1573a3418de44b4ab5c8415917800d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:17 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:45460
I20260924 20:11:17.992089 16794 tablet_service.cc:1514] Processing DeleteTablet for tablet 1573a3418de44b4ab5c8415917800d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:17 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:51554
I20260924 20:11:17.992586 16894 tablet_replica.cc:330] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: stopping tablet replica
I20260924 20:11:17.992646 16893 tablet_replica.cc:330] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: stopping tablet replica
I20260924 20:11:17.992803 16893 raft_consensus.cc:2229] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:11:17.992808 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 16440
I20260924 20:11:17.992941 16894 raft_consensus.cc:2229] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:11:17.993001 16893 raft_consensus.cc:2258] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:17.993377 16894 raft_consensus.cc:2258] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:17.994804 16894 ts_tablet_manager.cc:1905] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:17.996199 16895 tablet_replica.cc:330] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: stopping tablet replica
I20260924 20:11:17.996598 16895 raft_consensus.cc:2229] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:11:17.996986 16895 raft_consensus.cc:2258] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:17.997478 16894 ts_tablet_manager.cc:1918] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:17.997630 16894 log.cc:1196] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/1573a3418de44b4ab5c8415917800d31
I20260924 20:11:17.998016 16894 ts_tablet_manager.cc:1939] T 1573a3418de44b4ab5c8415917800d31 P 9b025e00fa64475f8a2c69cbc8bc357f: Deleting consensus metadata
I20260924 20:11:17.998740 16895 ts_tablet_manager.cc:1905] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:17.999279 16386 catalog_manager.cc:4841] TS 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647): tablet 1573a3418de44b4ab5c8415917800d31 (table test-table [id=72ee17368d874e1b8272c3052dbe14d6]) successfully deleted
I20260924 20:11:18.001144 16895 ts_tablet_manager.cc:1918] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:18.001262 16895 log.cc:1196] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/1573a3418de44b4ab5c8415917800d31
I20260924 20:11:18.001575 16895 ts_tablet_manager.cc:1939] T 1573a3418de44b4ab5c8415917800d31 P 123fb46fff0447a291547696c9dd9d9e: Deleting consensus metadata
W20260924 20:11:18.002067 16386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:40017 (error 108)
I20260924 20:11:18.002272 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 16594
I20260924 20:11:18.004159 16387 catalog_manager.cc:4841] TS 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561): tablet 1573a3418de44b4ab5c8415917800d31 (table test-table [id=72ee17368d874e1b8272c3052dbe14d6]) successfully deleted
I20260924 20:11:18.004740 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:18.004355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:40017 (local address 127.0.0.1:45472)
0924 20:11:18.004545 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:18.004633 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 20:11:18.004938 16386 catalog_manager.cc:4584] TS 84bcf6925e1b4e99ae523c7de879803c (127.0.115.129:40017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1573a3418de44b4ab5c8415917800d31: Network error: Client connection negotiation failed: client connection to 127.0.115.129:40017: connect: Connection refused (error 111)
I20260924 20:11:18.009364 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 16730
I20260924 20:11:18.015264 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 16369
I20260924 20:11:38.022239 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:43451
--webserver_interface=127.0.115.190
--webserver_port=34743
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:38.134941 16897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:38.135221 16897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:38.135263 16897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:38.139329 16897 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:11:38.139379 16897 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:11:38.139395 16897 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:11:38.139410 16897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:11:38.139426 16897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:38.139444 16897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:38.139460 16897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:11:38.139475 16897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:11:38.144701 16897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:43451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:38.145709 16897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:38.146692 16897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:38.155838 16902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.155874 16897 server_base.cc:1034] running on GCE node
W20260924 20:11:38.155864 16903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:38.155838 16905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.156657 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:38.157802 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:38.159021 16897 hybrid_clock.cc:648] HybridClock initialized: now 1790280698158991 us; error 48 us; skew 500 ppm
Sep 24 20:11:38 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:38.168027 16897 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:11:38.169116 16897 webserver.cc:460] Webserver started at http://127.0.115.190:34743/ using document root <none> and password file <none>
I20260924 20:11:38.169770 16897 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:38.169869 16897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:38.173945 16897 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:11:38.176266 16912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.177531 16897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:11:38.177691 16897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "ba9770723fb049ce916f20faec01529f"
format_stamp: "Formatted at 2026-09-24 20:10:55 on dist-test-slave-2pm7"
server_key: "6cacbc369e2140fc9007f1cf5225b920"
server_key_iv: "f38ef47ce39aa64b740be75e7ff2b714"
server_key_version: "encryptionkey@0"
I20260924 20:11:38.178308 16897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:38.193357 16897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:38.196841 16897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:38.197059 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:38.204739 16897 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:43451
I20260924 20:11:38.205024 16964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:43451 every 8 connection(s)
I20260924 20:11:38.205904 16897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:11:38.209090 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16897
I20260924 20:11:38.209528 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:40017
--local_ip_for_outbound_sockets=127.0.115.129
--tserver_master_addrs=127.0.115.190:43451
--webserver_port=38063
--webserver_interface=127.0.115.129
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:11:38.210747 16965 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:11:38.212611 16965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Bootstrap starting.
I20260924 20:11:38.230611 16965 log.cc:824] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:38.242072 16965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:11:38.242483 16965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: Bootstrap complete.
I20260924 20:11:38.246628 16965 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:38.247097 16965 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9770723fb049ce916f20faec01529f, State: Initialized, Role: FOLLOWER
I20260924 20:11:38.247659 16965 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:38.247814 16965 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:11:38.247903 16965 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:11:38.248052 16965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:11:38.249976 16965 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:38.250317 16965 leader_election.cc:302] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba9770723fb049ce916f20faec01529f; no voters:
I20260924 20:11:38.250651 16965 leader_election.cc:288] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 20:11:38.250795 16969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:11:38.251051 16969 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [term 3 LEADER]: Becoming Leader. State: Replica: ba9770723fb049ce916f20faec01529f, State: Running, Role: LEADER
I20260924 20:11:38.251439 16969 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } }
I20260924 20:11:38.251886 16965 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:11:38.252892 16970 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ba9770723fb049ce916f20faec01529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } } }
I20260924 20:11:38.253041 16970 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:38.253866 16971 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9770723fb049ce916f20faec01529f. Latest consensus state: current_term: 3 leader_uuid: "ba9770723fb049ce916f20faec01529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9770723fb049ce916f20faec01529f" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 43451 } } }
I20260924 20:11:38.253996 16971 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:38.256750 16986 catalog_manager.cc:1259] Loaded cluster ID: ef8648ae908e4845a1fbd3eeb04aa781
I20260924 20:11:38.256824 16986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: loading cluster ID for follower catalog manager: success
I20260924 20:11:38.258605 16986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba9770723fb049ce916f20faec01529f: acquiring CA information for follower catalog manager: success
F20260924 20:11:38.259534 16986 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f08dabe79ad google::LogMessage::Fail() at ??:0
@ 0x7f08dabea88a google::LogMessage::SendToLog() at ??:0
@ 0x7f08dabe74de google::LogMessage::Flush() at ??:0
@ 0x7f08dabe8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f08dd8ab18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f08e5550989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f08e552d023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f08e552d54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f08e5526652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f08e55259e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f08e556c541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f08dce2a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f08de46e6db start_thread at ??:0
@ 0x7f08d96e561f clone at ??:0
W20260924 20:11:38.351214 16967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:38.351480 16967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:38.351528 16967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:38.355785 16967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:38.355852 16967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:38.355943 16967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:11:38.361377 16967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:40017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=38063
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:38.362496 16967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:38.363569 16967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:38.374773 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.374934 16967 server_base.cc:1034] running on GCE node
W20260924 20:11:38.374784 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:38.374773 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.375545 16967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:38.376327 16967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:38.377525 16967 hybrid_clock.cc:648] HybridClock initialized: now 1790280698377506 us; error 38 us; skew 500 ppm
Sep 24 20:11:38 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:38.380439 16967 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:11:38.381429 16967 webserver.cc:460] Webserver started at http://127.0.115.129:38063/ using document root <none> and password file <none>
I20260924 20:11:38.382076 16967 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:38.382182 16967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:38.386394 16967 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 20:11:38.388845 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.390390 16967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:11:38.390547 16967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "84bcf6925e1b4e99ae523c7de879803c"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "116d7c88e3859e61ff837ab3b6e049e2"
server_key_iv: "075c8610c300f61be36ec91ec00781d9"
server_key_version: "encryptionkey@0"
I20260924 20:11:38.391155 16967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:38.409770 16967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:38.413362 16967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:38.413589 16967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:38.414273 16967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:38.415467 17023 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:11:38.421270 16967 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:11:38.421373 16967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 20:11:38.421470 16967 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:11:38.423280 16967 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:11:38.423355 16967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 20:11:38.423672 17023 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap starting.
I20260924 20:11:38.435329 16967 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:40017
I20260924 20:11:38.436493 17130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:40017 every 8 connection(s)
I20260924 20:11:38.436739 16967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:11:38.436991 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 16967
I20260924 20:11:38.437536 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:35647
--local_ip_for_outbound_sockets=127.0.115.130
--tserver_master_addrs=127.0.115.190:43451
--webserver_port=39655
--webserver_interface=127.0.115.130
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:11:38.448333 17023 log.cc:824] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:38.458621 17023 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:11:38.459106 17023 tablet_bootstrap.cc:492] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Bootstrap complete.
I20260924 20:11:38.460717 17023 ts_tablet_manager.cc:1397] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.010s
I20260924 20:11:38.464442 17023 raft_consensus.cc:348] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:38.465106 17023 raft_consensus.cc:729] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcf6925e1b4e99ae523c7de879803c, State: Initialized, Role: FOLLOWER
I20260924 20:11:38.465760 17023 consensus_queue.cc:260] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:38.467296 17023 ts_tablet_manager.cc:1428] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260924 20:11:38.473616 17134 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:11:38.440251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:50311)
0924 20:11:38.441379 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.441440 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:38.442286 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:38.472607 (+ 30321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":172,"threads_started":1}
W20260924 20:11:38.474259 17131 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:43451 (0 consecutive failures): Network error: Failed to ping master at 127.0.115.190:43451: Client connection negotiation failed: client connection to 127.0.115.190:43451: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:11:38.475109 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.474783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:54351)
0924 20:11:38.474938 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.475022 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 20:11:38.475880 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.475624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:52881)
0924 20:11:38.475747 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.475803 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:11:38.476199 17131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:11:38.564944 17133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:38.565213 17133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:38.565256 17133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:38.569173 17133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:38.569283 17133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:38.569383 17133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:11:38.574842 17133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:35647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=39655
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:38.575923 17133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:38.577032 17133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:38.587114 17144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.587339 17133 server_base.cc:1034] running on GCE node
W20260924 20:11:38.587114 17141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:38.587142 17142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.587878 17133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:38.588588 17133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:38.589751 17133 hybrid_clock.cc:648] HybridClock initialized: now 1790280698589717 us; error 46 us; skew 500 ppm
Sep 24 20:11:38 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:38.592520 17133 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:11:38.593418 17133 webserver.cc:460] Webserver started at http://127.0.115.130:39655/ using document root <none> and password file <none>
I20260924 20:11:38.593961 17133 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:38.594020 17133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:38.598095 17133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:11:38.600168 17151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.601519 17133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:11:38.601670 17133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "9b025e00fa64475f8a2c69cbc8bc357f"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "85c76b031d1c6a2115030ac7d5542034"
server_key_iv: "e013cf0c319ca9118bbdfc331245a0c3"
server_key_version: "encryptionkey@0"
I20260924 20:11:38.602217 17133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:38.628468 17133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:38.631924 17133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:38.632164 17133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:38.632789 17133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:38.633644 17133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:38.633690 17133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.633752 17133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:38.633785 17133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.642872 17133 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:35647
I20260924 20:11:38.642903 17264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:35647 every 8 connection(s)
I20260924 20:11:38.643868 17133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:11:38.647691 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.646145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:37999)
0924 20:11:38.646856 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.646965 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":113,"threads_started":1}
W20260924 20:11:38.648315 17265 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:43451 (0 consecutive failures): Network error: Failed to ping master at 127.0.115.190:43451: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
I20260924 20:11:38.648851 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 17133
I20260924 20:11:38.649044 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.648757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:50629)
0924 20:11:38.648892 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.648984 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 20:11:38.649364 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:46561
--local_ip_for_outbound_sockets=127.0.115.131
--tserver_master_addrs=127.0.115.190:43451
--webserver_port=40023
--webserver_interface=127.0.115.131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:11:38.649780 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.649567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:38829)
0924 20:11:38.649686 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.649731 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 20:11:38.650017 17265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:11:38.759166 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:38.759408 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:38.759449 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:38.763278 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:38.763330 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:38.763384 17268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:11:38.768443 17268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:46561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=40023
--tserver_master_addrs=127.0.115.190:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:38.769371 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:38.770308 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:38.780179 17273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:38.780267 17276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.780247 17268 server_base.cc:1034] running on GCE node
W20260924 20:11:38.780184 17274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:38.780745 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:38.781363 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:38.782543 17268 hybrid_clock.cc:648] HybridClock initialized: now 1790280698782513 us; error 50 us; skew 500 ppm
Sep 24 20:11:38 dist-test-slave-2pm7 krb5kdc[15823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:38.785689 17268 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:11:38.786682 17268 webserver.cc:460] Webserver started at http://127.0.115.131:40023/ using document root <none> and password file <none>
I20260924 20:11:38.787210 17268 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:38.787269 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:38.791209 17268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:11:38.793310 17283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.794559 17268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:11:38.794662 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "123fb46fff0447a291547696c9dd9d9e"
format_stamp: "Formatted at 2026-09-24 20:10:56 on dist-test-slave-2pm7"
server_key: "9ee2f550446864111344560488d561b3"
server_key_iv: "02f4e05f2c4964b7919e90251e613e32"
server_key_version: "encryptionkey@0"
I20260924 20:11:38.795202 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:38.815845 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:38.818763 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:38.818965 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:38.819576 17268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:38.820492 17268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:38.820560 17268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.820632 17268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:38.820672 17268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:38.830193 17268 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:46561
I20260924 20:11:38.830256 17396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:46561 every 8 connection(s)
I20260924 20:11:38.831259 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:11:38.834980 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.833499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:45243)
0924 20:11:38.834176 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.834288 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":108,"threads_started":1}
W20260924 20:11:38.835420 17397 heartbeater.cc:644] Failed to heartbeat to 127.0.115.190:43451 (0 consecutive failures): Network error: Failed to ping master at 127.0.115.190:43451: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
I20260924 20:11:38.836053 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 17268
I20260924 20:11:38.836081 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.835791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:48137)
0924 20:11:38.835920 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.836010 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 20:11:38.836733 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:38.836544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:60201)
0924 20:11:38.836649 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:38.836689 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:11:38.836946 17397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:11:39.477325 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:39.476850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:47755)
0924 20:11:39.477110 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:39.477176 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260924 20:11:39.650929 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:39.650540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:40529)
0924 20:11:39.650783 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:39.650831 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260924 20:11:39.837911 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:39.837493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:54855)
0924 20:11:39.837755 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:39.837809 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":66,"threads_started":1}
I20260924 20:11:40.041522 17404 raft_consensus.cc:482] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:40.041730 17404 raft_consensus.cc:504] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:40.042886 17404 leader_election.cc:288] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561), 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
Sep 24 20:11:40 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.130@KRBTEST.COM
Sep 24 20:11:40 dist-test-slave-2pm7 krb5kdc[15823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.131@KRBTEST.COM
I20260924 20:11:40.059890 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:40.043074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:46561 (local address 127.0.115.129:56139)
0924 20:11:40.043349 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:40.043367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:40.043481 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:40.045404 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:40.045410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:40.046001 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:40.046525 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:40.046544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.048527 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.048531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:40.050174 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:40.050186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.051234 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.051238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:40.051507 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:40.052437 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:40.052466 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:40.054853 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:40.057242 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:40.057247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:40.057249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:40.057641 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:40.058075 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:40.058078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:40.058079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:40.058188 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:40.058882 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:40.058888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:40.059131 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 20:11:40.059331 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 20:11:40.059682 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":245,"spinlock_wait_cycles":40960,"thread_start_us":86,"threads_started":1}
I20260924 20:11:40.059908 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:40.043611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:35925 (local address 127.0.115.130:35647)
0924 20:11:40.044015 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:40.044036 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:40.044070 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:11:40.044710 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:40.044715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:40.045017 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:40.045338 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:40.046731 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.048348 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.050327 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.051094 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.051326 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:40.054804 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:40.054811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:40.054832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:40.054868 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:40.056757 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:40.057778 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:40.057784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:40.057792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:40.057914 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:40.058307 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:40.058311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:40.058312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:40.058756 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:40.058893 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:40.059443 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 20:11:40.059701 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":88,"threads_started":1}
I20260924 20:11:40.059966 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:40.043356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:56139 (local address 127.0.115.131:46561)
0924 20:11:40.043779 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:40.043806 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:40.043855 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:11:40.044645 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:40.044651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:40.045018 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:40.045337 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:40.046709 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.048349 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.050327 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.051095 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.051289 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:40.055042 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:40.055046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:40.055057 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:40.055085 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:40.057064 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:40.057802 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:40.057810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:40.057818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:40.057946 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:40.058346 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:40.058348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:40.058350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:40.058750 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:40.058893 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:40.059451 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 20:11:40.059736 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":65,"threads_started":1}
I20260924 20:11:40.059962 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:40.043296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:35647 (local address 127.0.115.129:35925)
0924 20:11:40.043611 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:40.043627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:40.043738 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:40.045406 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:40.045412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:40.045990 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:40.046524 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:40.046542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.048527 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.048531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:40.050174 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:40.050186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:40.051282 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:40.051285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:40.051507 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:40.052437 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:40.052466 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:40.054495 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:40.057151 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:40.057159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:40.057176 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:40.057639 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:40.058028 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:40.058033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:40.058035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:40.058200 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:40.058884 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:40.058890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:40.059134 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 20:11:40.059331 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 20:11:40.059768 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":394,"spinlock_wait_cycles":46464,"thread_start_us":73,"threads_started":1}
W20260924 20:11:40.063184 17018 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
W20260924 20:11:40.063369 17017 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
I20260924 20:11:40.063495 17017 leader_election.cc:302] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84bcf6925e1b4e99ae523c7de879803c; no voters: 123fb46fff0447a291547696c9dd9d9e, 9b025e00fa64475f8a2c69cbc8bc357f
I20260924 20:11:40.063817 17404 raft_consensus.cc:2726] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:11:40.478492 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:40.478158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:36625)
0924 20:11:40.478334 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:40.478400 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 20:11:40.652299 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:40.651822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:38203)
0924 20:11:40.652136 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:40.652199 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":80,"threads_started":1}
I20260924 20:11:40.839164 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:40.838701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:52335)
0924 20:11:40.839001 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:40.839070 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20260924 20:11:41.479894 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:41.479453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:49391)
0924 20:11:41.479727 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:41.479792 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260924 20:11:41.653676 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:41.653219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:53113)
0924 20:11:41.653514 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:41.653577 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":76,"threads_started":1}
I20260924 20:11:41.691642 17413 raft_consensus.cc:482] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:41.691776 17413 raft_consensus.cc:504] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:41.692214 17413 leader_election.cc:288] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561), 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
W20260924 20:11:41.693162 17018 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
W20260924 20:11:41.693349 17017 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
I20260924 20:11:41.693426 17017 leader_election.cc:302] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84bcf6925e1b4e99ae523c7de879803c; no voters: 123fb46fff0447a291547696c9dd9d9e, 9b025e00fa64475f8a2c69cbc8bc357f
I20260924 20:11:41.693554 17413 raft_consensus.cc:2726] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:11:41.840500 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:41.840020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:35173)
0924 20:11:41.840353 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:41.840411 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260924 20:11:42.481245 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:42.480756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:34519)
0924 20:11:42.481054 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:42.481134 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260924 20:11:42.654994 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:42.654556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:58711)
0924 20:11:42.654846 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:42.654902 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
I20260924 20:11:42.841863 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:42.841425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:54687)
0924 20:11:42.841701 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:42.841766 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260924 20:11:43.483294 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:43.482576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:35875)
0924 20:11:43.483033 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:43.483137 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":114,"threads_started":1}
I20260924 20:11:43.656301 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:43.655823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:53037)
0924 20:11:43.656132 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:43.656199 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":131,"threads_started":1}
I20260924 20:11:43.843214 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:43.842728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:49463)
0924 20:11:43.843013 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:43.843080 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260924 20:11:43.925541 17421 raft_consensus.cc:482] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:43.925699 17421 raft_consensus.cc:504] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:43.926107 17421 leader_election.cc:288] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561), 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
W20260924 20:11:43.927078 17018 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
W20260924 20:11:43.927193 17017 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
I20260924 20:11:43.927244 17017 leader_election.cc:302] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84bcf6925e1b4e99ae523c7de879803c; no voters: 123fb46fff0447a291547696c9dd9d9e, 9b025e00fa64475f8a2c69cbc8bc357f
I20260924 20:11:43.927424 17421 raft_consensus.cc:2726] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:11:44.484580 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:44.484149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:49651)
0924 20:11:44.484419 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:44.484478 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260924 20:11:44.657598 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:44.657113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:47755)
0924 20:11:44.657451 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:44.657502 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":142,"threads_started":1}
I20260924 20:11:44.844520 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:44.844067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:39313)
0924 20:11:44.844364 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:44.844426 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260924 20:11:45.485982 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:45.485444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:60309)
0924 20:11:45.485756 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:45.485856 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
I20260924 20:11:45.658968 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:45.658467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:59635)
0924 20:11:45.658794 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:45.658867 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
I20260924 20:11:45.764734 17427 raft_consensus.cc:482] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:45.764869 17427 raft_consensus.cc:504] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:45.765252 17427 leader_election.cc:288] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561), 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
W20260924 20:11:45.766065 17018 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
W20260924 20:11:45.766161 17017 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
I20260924 20:11:45.766273 17017 leader_election.cc:302] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84bcf6925e1b4e99ae523c7de879803c; no voters: 123fb46fff0447a291547696c9dd9d9e, 9b025e00fa64475f8a2c69cbc8bc357f
I20260924 20:11:45.766435 17427 raft_consensus.cc:2726] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:11:45.845986 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:45.845434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:51789)
0924 20:11:45.845809 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:45.845882 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":95,"threads_started":1}
I20260924 20:11:46.487388 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:46.486904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:55147)
0924 20:11:46.487214 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:46.487283 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260924 20:11:46.660259 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:46.659804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:55807)
0924 20:11:46.660108 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:46.660164 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":85,"threads_started":1}
I20260924 20:11:46.847285 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:46.846856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:32939)
0924 20:11:46.847128 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:46.847187 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260924 20:11:47.488802 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:47.488264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:56095)
0924 20:11:47.488572 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:47.488650 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260924 20:11:47.661487 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:47.661039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:53733)
0924 20:11:47.661321 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:47.661384 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260924 20:11:47.848616 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:47.848139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:33027)
0924 20:11:47.848452 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:47.848515 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260924 20:11:48.490295 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:48.489815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:59199)
0924 20:11:48.490125 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:48.490191 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260924 20:11:48.662873 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:48.662418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:41833)
0924 20:11:48.662696 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:48.662760 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260924 20:11:48.849931 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:48.849480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:42999)
0924 20:11:48.849754 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:48.849830 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260924 20:11:49.491811 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:49.491343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:52873)
0924 20:11:49.491628 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:49.491704 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260924 20:11:49.664300 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:49.663743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:38631)
0924 20:11:49.664123 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:49.664188 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":93,"threads_started":1}
I20260924 20:11:49.851207 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:49.850747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:35627)
0924 20:11:49.851041 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:49.851105 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260924 20:11:50.493100 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:50.492654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:35441)
0924 20:11:50.492934 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:50.492998 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260924 20:11:50.665490 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:50.665082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:47705)
0924 20:11:50.665328 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:50.665383 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260924 20:11:50.852505 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:50.852078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:59139)
0924 20:11:50.852334 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:50.852394 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260924 20:11:51.494410 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:51.493871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:37541)
0924 20:11:51.494151 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:51.494223 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260924 20:11:51.666710 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:51.666258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:48263)
0924 20:11:51.666554 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:51.666613 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":73,"threads_started":1}
I20260924 20:11:51.853700 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:51.853298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:45993)
0924 20:11:51.853549 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:51.853603 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260924 20:11:52.333896 17447 raft_consensus.cc:482] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:52.334041 17447 raft_consensus.cc:504] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123fb46fff0447a291547696c9dd9d9e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46561 } } peers { permanent_uuid: "9b025e00fa64475f8a2c69cbc8bc357f" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 35647 } } peers { permanent_uuid: "84bcf6925e1b4e99ae523c7de879803c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 40017 } }
I20260924 20:11:52.334537 17447 leader_election.cc:288] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561), 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647)
W20260924 20:11:52.335361 17018 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 123fb46fff0447a291547696c9dd9d9e (127.0.115.131:46561): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
W20260924 20:11:52.335479 17017 leader_election.cc:341] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b025e00fa64475f8a2c69cbc8bc357f (127.0.115.130:35647): Not found: Tablet not found: 1573a3418de44b4ab5c8415917800d31
I20260924 20:11:52.335546 17017 leader_election.cc:302] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84bcf6925e1b4e99ae523c7de879803c; no voters: 123fb46fff0447a291547696c9dd9d9e, 9b025e00fa64475f8a2c69cbc8bc357f
I20260924 20:11:52.335700 17447 raft_consensus.cc:2726] T 1573a3418de44b4ab5c8415917800d31 P 84bcf6925e1b4e99ae523c7de879803c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:11:52.495874 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:52.495323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:42421)
0924 20:11:52.495666 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:52.495739 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
I20260924 20:11:52.668126 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:52.667641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:48843)
0924 20:11:52.667962 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:52.668028 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
I20260924 20:11:52.855000 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:52.854551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.131:41273)
0924 20:11:52.854831 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:52.854893 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260924 20:11:53.497252 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:53.496843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.129:44491)
0924 20:11:53.497092 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:53.497155 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260924 20:11:53.669715 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:53.669244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:43451 (local address 127.0.115.130:34107)
0924 20:11:53.669544 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:53.669613 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.190:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260924 20:11:53.837323 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 16967
I20260924 20:11:53.843909 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 17133
I20260924 20:11:53.849704 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 17268
2026-09-24T20:11:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59351 ms)
[----------] 30 tests from SecurityITest (247308 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:11:53 dist-test-slave-2pm7 krb5kdc[17455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:11:53 dist-test-slave-2pm7 krb5kdc[17455](info): set up 2 sockets
Sep 24 20:11:53 dist-test-slave-2pm7 krb5kdc[17455](info): commencing operation
krb5kdc: starting...
W20260924 20:11:54.887926 462 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:11:54 dist-test-slave-2pm7 krb5kdc[17455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:11:55.015455 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:41271
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:41271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:55.128881 17468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:55.129185 17468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:55.129227 17468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:55.133306 17468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:11:55.133363 17468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:55.133383 17468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:55.133399 17468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:11:55.133414 17468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:11:55.138830 17468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:41271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:41271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:55.139981 17468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:55.141104 17468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:55.150118 17474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:55.150125 17476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:55.150105 17473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:55.150175 17468 server_base.cc:1034] running on GCE node
I20260924 20:11:55.151099 17468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:55.152343 17468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:55.153512 17468 hybrid_clock.cc:648] HybridClock initialized: now 1790280715153496 us; error 24 us; skew 500 ppm
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:55.156301 17468 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:11:55.157217 17468 webserver.cc:460] Webserver started at http://127.0.115.190:33207/ using document root <none> and password file <none>
I20260924 20:11:55.157838 17468 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:55.157935 17468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:55.158192 17468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:55.160445 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "d865561f56ee4344b04c379098970e40"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "7a78582ce0a8969429b364d204963f33"
server_key_iv: "08bca63c9d34ac2a799096e33a411bbb"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.161056 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "d865561f56ee4344b04c379098970e40"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "7a78582ce0a8969429b364d204963f33"
server_key_iv: "08bca63c9d34ac2a799096e33a411bbb"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.165163 17468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 20:11:55.167843 17483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.169274 17468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:11:55.169456 17468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "d865561f56ee4344b04c379098970e40"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "7a78582ce0a8969429b364d204963f33"
server_key_iv: "08bca63c9d34ac2a799096e33a411bbb"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.169593 17468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:55.198105 17468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:55.201089 17468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:55.201313 17468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:55.208848 17468 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:41271
I20260924 20:11:55.208920 17535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:41271 every 8 connection(s)
I20260924 20:11:55.209867 17468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:11:55.212291 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 17468
I20260924 20:11:55.212436 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:11:55.212725 462 external_mini_cluster.cc:1428] Setting key 50527206ca82bcbe03994ef82ebc1519
I20260924 20:11:55.212975 17536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:55.219835 17536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40: Bootstrap starting.
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:55.222667 17536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:55.223569 17536 log.cc:824] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:55.225965 17536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40: No bootstrap required, opened a new log
I20260924 20:11:55.228065 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.214198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.0.115.190:41271)
0924 20:11:55.214688 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:55.214704 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:55.214741 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:11:55.215564 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:55.215584 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:55.215917 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:55.216398 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:55.217169 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.218296 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.219203 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.219705 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.219895 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.222697 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:55.222732 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:55.222755 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:55.222800 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:55.224691 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.225222 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.225227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.225233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.225364 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.225670 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.225673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.225674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.226131 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:55.226322 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:55.226637 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 20:11:55.227011 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":162,"threads_started":1}
I20260924 20:11:55.229552 17536 raft_consensus.cc:348] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d865561f56ee4344b04c379098970e40" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41271 } }
I20260924 20:11:55.229835 17536 raft_consensus.cc:374] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:55.229883 17536 raft_consensus.cc:729] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d865561f56ee4344b04c379098970e40, State: Initialized, Role: FOLLOWER
I20260924 20:11:55.230340 17536 consensus_queue.cc:260] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d865561f56ee4344b04c379098970e40" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41271 } }
I20260924 20:11:55.230515 17536 raft_consensus.cc:388] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:11:55.230618 17536 raft_consensus.cc:482] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:11:55.230738 17536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:55.231686 17536 raft_consensus.cc:504] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d865561f56ee4344b04c379098970e40" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41271 } }
I20260924 20:11:55.232095 17536 leader_election.cc:302] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d865561f56ee4344b04c379098970e40; no voters:
I20260924 20:11:55.232412 17536 leader_election.cc:288] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:11:55.232599 17541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:55.232937 17541 raft_consensus.cc:686] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [term 1 LEADER]: Becoming Leader. State: Replica: d865561f56ee4344b04c379098970e40, State: Running, Role: LEADER
I20260924 20:11:55.233235 17541 consensus_queue.cc:237] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d865561f56ee4344b04c379098970e40" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41271 } }
I20260924 20:11:55.233800 17536 sys_catalog.cc:564] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:11:55.234929 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d865561f56ee4344b04c379098970e40. Latest consensus state: current_term: 1 leader_uuid: "d865561f56ee4344b04c379098970e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d865561f56ee4344b04c379098970e40" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41271 } } }
I20260924 20:11:55.235064 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:55.234989 17542 sys_catalog.cc:455] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d865561f56ee4344b04c379098970e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d865561f56ee4344b04c379098970e40" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 41271 } } }
I20260924 20:11:55.235275 17542 sys_catalog.cc:458] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40 [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:55.236346 17550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:11:55.239631 17550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:11:55.245725 17550 catalog_manager.cc:1347] Generated new cluster ID: 9e39e635973447a5b17717c4767c2ba7
I20260924 20:11:55.245800 17550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:11:55.254833 17550 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:11:55.256111 17550 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:11:55.262878 17550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40: Generated new TSK 0
I20260924 20:11:55.263749 17550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:11:55.298870 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41271
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:55.418394 17562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:55.418727 17562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:55.418802 17562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:55.422901 17562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:55.423031 17562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:55.423123 17562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:11:55.428718 17562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:55.429914 17562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:55.431032 17562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:55.441581 17567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:55.441572 17562 server_base.cc:1034] running on GCE node
W20260924 20:11:55.441620 17568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:55.441546 17570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:55.442237 17562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:55.442932 17562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:55.444098 17562 hybrid_clock.cc:648] HybridClock initialized: now 1790280715444078 us; error 29 us; skew 500 ppm
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:55.447075 17562 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:11:55.448114 17562 webserver.cc:460] Webserver started at http://127.0.115.129:42623/ using document root <none> and password file <none>
I20260924 20:11:55.448709 17562 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:55.448800 17562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:55.449025 17562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:55.451113 17562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "af77514638544809817d7f3e23bd59c5"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "00572d704b94f379c90620cba1d1ecc2"
server_key_iv: "ac83803cf5a2cb947d95aa94e5105e9f"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.451733 17562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "af77514638544809817d7f3e23bd59c5"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "00572d704b94f379c90620cba1d1ecc2"
server_key_iv: "ac83803cf5a2cb947d95aa94e5105e9f"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.455788 17562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 20:11:55.458346 17577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.459642 17562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:11:55.459784 17562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "af77514638544809817d7f3e23bd59c5"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "00572d704b94f379c90620cba1d1ecc2"
server_key_iv: "ac83803cf5a2cb947d95aa94e5105e9f"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.459908 17562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:55.479539 17562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:55.482959 17562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:55.483187 17562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:55.484059 17562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:55.485328 17562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:55.485397 17562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.485461 17562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:55.485525 17562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 20:11:55.495644 17562 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:39269
I20260924 20:11:55.495668 17690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:39269 every 8 connection(s)
I20260924 20:11:55.496615 17562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:55.506232 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 17562
I20260924 20:11:55.506384 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:11:55.506757 462 external_mini_cluster.cc:1428] Setting key 2a7d075a61bed953e32c0ae18bfbc6e8
I20260924 20:11:55.510519 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.498424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:58381 (local address 127.0.115.190:41271)
0924 20:11:55.498604 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:55.498609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:55.499468 (+ 859us) server_negotiation.cc:407] Connection header received
0924 20:11:55.500447 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:55.500451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:55.500498 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:55.500587 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:55.501765 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.502298 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.503430 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.503657 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.503735 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.506473 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:55.506480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:55.506485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:55.506522 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:55.508172 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.508775 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.508810 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.508814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.508914 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.509269 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.509276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.509279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.509552 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:55.509666 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:55.510211 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 20:11:55.510349 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:11:55.511432 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.498858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41271 (local address 127.0.115.129:58381)
0924 20:11:55.499303 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:55.499347 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:55.500206 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:55.500746 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:55.500753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:55.501114 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:55.501623 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:55.501637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.502417 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.502421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:55.503322 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:55.503331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.503795 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.503798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:55.504009 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.504584 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:55.504601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:55.506205 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:55.508299 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:55.508305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:55.508324 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:55.508663 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:55.509019 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:55.509022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:55.509024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:55.509169 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:55.509664 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:55.509669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:55.509932 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 20:11:55.510168 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 20:11:55.510546 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260924 20:11:55.512866 17691 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41271
I20260924 20:11:55.513154 17691 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:55.513690 17691 heartbeater.cc:511] Master 127.0.115.190:41271 requested a full tablet report, sending...
I20260924 20:11:55.515873 17500 ts_manager.cc:194] Registered new tserver with Master: af77514638544809817d7f3e23bd59c5 (127.0.115.129:39269)
I20260924 20:11:55.517854 17500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:58381
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:11:55.539417 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41271
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:55.654821 17696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:55.655104 17696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:55.655148 17696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:55.659273 17696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:55.659339 17696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:55.659405 17696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:11:55.664808 17696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:55.665899 17696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:55.666910 17696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:55.676654 17701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:55.676769 17696 server_base.cc:1034] running on GCE node
W20260924 20:11:55.676674 17704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:55.676674 17702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:55.677357 17696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:55.678014 17696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:55.679198 17696 hybrid_clock.cc:648] HybridClock initialized: now 1790280715679177 us; error 39 us; skew 500 ppm
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:55.681964 17696 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:11:55.682786 17696 webserver.cc:460] Webserver started at http://127.0.115.130:37857/ using document root <none> and password file <none>
I20260924 20:11:55.683344 17696 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:55.683434 17696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:55.683669 17696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:55.685755 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "1c21e5c87ef845f8af4f86d7e2b3653c"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "3cba814ea547420c2934cb09add368ec"
server_key_iv: "ab8b5088bb4dc8736f2c771210c13dac"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.686313 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "1c21e5c87ef845f8af4f86d7e2b3653c"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "3cba814ea547420c2934cb09add368ec"
server_key_iv: "ab8b5088bb4dc8736f2c771210c13dac"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.690047 17696 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:11:55.692453 17711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.693670 17696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:11:55.693799 17696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "1c21e5c87ef845f8af4f86d7e2b3653c"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "3cba814ea547420c2934cb09add368ec"
server_key_iv: "ab8b5088bb4dc8736f2c771210c13dac"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.693907 17696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:55.706621 17696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:55.709547 17696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:55.709746 17696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:55.710315 17696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:55.711151 17696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:55.711222 17696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.711292 17696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:55.711342 17696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.720434 17696 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:43787
I20260924 20:11:55.720463 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:43787 every 8 connection(s)
I20260924 20:11:55.721331 17696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:11:55.726971 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 17696
I20260924 20:11:55.727098 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:11:55.727389 462 external_mini_cluster.cc:1428] Setting key 1690ab648f6d6826031ee12387f942c6
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:55.736006 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.723145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:38199 (local address 127.0.115.190:41271)
0924 20:11:55.723292 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:55.723297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:55.724150 (+ 853us) server_negotiation.cc:407] Connection header received
0924 20:11:55.725381 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:55.725384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:55.725428 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:55.725505 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:55.726637 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.727473 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.728939 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.729188 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.729310 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.731819 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:55.731824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:55.731826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:55.731856 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:55.733396 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.733944 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.733949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.733951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.734039 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.734467 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.734472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.734474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.734673 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:55.734817 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:55.735331 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 20:11:55.735785 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:11:55.736712 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.723515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41271 (local address 127.0.115.130:38199)
0924 20:11:55.724002 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:55.724052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:55.725194 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:55.725670 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:55.725677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:55.726044 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:55.726513 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:55.726525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.727590 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.727593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:55.728805 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:55.728814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.729348 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.729352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:55.729538 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.730170 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:55.730187 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:55.731648 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:55.733522 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:55.733528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:55.733544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:55.733825 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:55.734139 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:55.734143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:55.734145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:55.734359 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:55.734797 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:55.734806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:55.735083 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 20:11:55.735288 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 20:11:55.735648 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
I20260924 20:11:55.737949 17825 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41271
I20260924 20:11:55.738229 17825 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:55.738752 17825 heartbeater.cc:511] Master 127.0.115.190:41271 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:11:55.739974 17500 ts_manager.cc:194] Registered new tserver with Master: 1c21e5c87ef845f8af4f86d7e2b3653c (127.0.115.130:43787)
I20260924 20:11:55.740547 17500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:38199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:11:55.755236 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41271
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:55.868474 17830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:55.868755 17830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:55.868798 17830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:55.873174 17830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:55.873234 17830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:55.873292 17830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:11:55.878665 17830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:55.879719 17830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:55.880894 17830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:55.890686 17836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:55.890733 17835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:55.891008 17830 server_base.cc:1034] running on GCE node
W20260924 20:11:55.890674 17838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:55.891517 17830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:55.892227 17830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:55.893414 17830 hybrid_clock.cc:648] HybridClock initialized: now 1790280715893392 us; error 41 us; skew 500 ppm
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:55.896277 17830 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:11:55.897224 17830 webserver.cc:460] Webserver started at http://127.0.115.131:33027/ using document root <none> and password file <none>
I20260924 20:11:55.897892 17830 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:55.897984 17830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:55.898197 17830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:55.900353 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "059370ae5ee94117ba5e26f22623e3e6"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "21ab1c4f85e5fb7a19a056c662c3eb53"
server_key_iv: "fc309680e598a12f20673396104b39a7"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.900947 17830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "059370ae5ee94117ba5e26f22623e3e6"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "21ab1c4f85e5fb7a19a056c662c3eb53"
server_key_iv: "fc309680e598a12f20673396104b39a7"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.904857 17830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:11:55.907372 17845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.908703 17830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:11:55.908843 17830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "059370ae5ee94117ba5e26f22623e3e6"
format_stamp: "Formatted at 2026-09-24 20:11:55 on dist-test-slave-2pm7"
server_key: "21ab1c4f85e5fb7a19a056c662c3eb53"
server_key_iv: "fc309680e598a12f20673396104b39a7"
server_key_version: "encryptionkey@0"
I20260924 20:11:55.908960 17830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:55.932871 17830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:55.936774 17830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:55.936957 17830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:55.937608 17830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:55.938571 17830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:55.938633 17830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.938678 17830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:55.938701 17830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:55.948385 17830 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:36775
I20260924 20:11:55.948468 17958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:36775 every 8 connection(s)
I20260924 20:11:55.949368 17830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:11:55.952014 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 17830
I20260924 20:11:55.952159 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:11:55.952518 462 external_mini_cluster.cc:1428] Setting key 0b813665afcfd150338a7cec48e9c179
Sep 24 20:11:55 dist-test-slave-2pm7 krb5kdc[17455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:55.963588 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.951335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:52941 (local address 127.0.115.190:41271)
0924 20:11:55.951519 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:55.951524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:55.952493 (+ 969us) server_negotiation.cc:407] Connection header received
0924 20:11:55.954100 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:55.954104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:55.954159 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:55.954234 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:55.955582 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.956181 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.957334 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.957562 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.957676 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.960281 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:55.960284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:55.960286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:55.960310 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:55.961660 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.962185 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.962188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.962190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.962262 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.962576 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.962578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.962579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.962749 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:55.962824 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:55.963301 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 20:11:55.963402 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 20:11:55.964581 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.951761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:41271 (local address 127.0.115.131:52941)
0924 20:11:55.952300 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:55.952365 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:55.953807 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:55.954430 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:55.954437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:55.954948 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:55.955441 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:55.955455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.956302 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.956305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:55.957202 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:55.957212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.957716 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.957720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:55.957915 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.958495 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:55.958512 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:55.960121 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:55.961790 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:55.961795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:55.961808 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:55.962093 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:55.962348 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:55.962351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:55.962352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:55.962493 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:55.962841 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:55.962846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:55.963092 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 20:11:55.963258 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 20:11:55.963610 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
I20260924 20:11:55.965984 17959 heartbeater.cc:347] Connected to a master server at 127.0.115.190:41271
I20260924 20:11:55.966298 17959 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:55.966956 17959 heartbeater.cc:511] Master 127.0.115.190:41271 requested a full tablet report, sending...
I20260924 20:11:55.968138 17499 ts_manager.cc:194] Registered new tserver with Master: 059370ae5ee94117ba5e26f22623e3e6 (127.0.115.131:36775)
I20260924 20:11:55.968756 17499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:52941
I20260924 20:11:55.978845 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:11:55.988495 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:55.980971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.0.115.190:41271)
0924 20:11:55.981138 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:55.981142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:55.981310 (+ 168us) server_negotiation.cc:407] Connection header received
0924 20:11:55.981481 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:55.981489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:55.981533 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:55.981621 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:55.982353 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.982884 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.983771 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:55.984054 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:55.984165 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:55.985530 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:55.985534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:55.985537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:55.985571 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:55.986758 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.987186 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.987190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.987192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.987277 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:55.987552 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:55.987556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:55.987558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:55.987793 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:55.987935 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:55.988200 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 20:11:55.988318 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:11:55.991472 17499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:11:55.994547 17499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:11:56.010300 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.004196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.0.115.131:36775)
0924 20:11:56.005738 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.005753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.005778 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:11:56.005858 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.005863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.006060 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.006209 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:56.006624 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.007448 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.008943 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.009552 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.009689 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.009870 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.010034 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.010135 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1437,"thread_start_us":247,"threads_started":1}
I20260924 20:11:56.010296 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.004077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:36775 (local address 127.0.0.1:59148)
0924 20:11:56.004949 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.005004 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:56.005253 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:56.006319 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:56.006322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:56.006336 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:56.006504 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.006511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.007624 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.007627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.008807 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.008818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.009690 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.009694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.009803 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.009840 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:11:56.009996 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 20:11:56.010134 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":61,"spinlock_wait_cycles":234880,"thread_start_us":92,"threads_started":1}
I20260924 20:11:56.010819 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.004157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:43787 (local address 127.0.0.1:34938)
0924 20:11:56.004993 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.005009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:56.005252 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:56.006114 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:56.006121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:56.006153 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:56.006343 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.006350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.007898 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.007901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.009174 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.009185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.010362 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.010366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.010497 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.010519 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:11:56.010582 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 20:11:56.010665 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":20,"spinlock_wait_cycles":4864,"thread_start_us":146,"threads_started":1}
I20260924 20:11:56.011055 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.005135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39269 (local address 127.0.0.1:36322)
0924 20:11:56.005509 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.005521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:56.005605 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:56.006303 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:56.006306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:56.006324 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:56.006485 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.006493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.008087 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.008090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.009219 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.009230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.010362 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.010366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.010757 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.010779 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:11:56.010861 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:11:56.010933 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":254,"thread_start_us":88,"threads_started":1}
I20260924 20:11:56.011448 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.004403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.115.130:43787)
0924 20:11:56.005587 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.005601 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.005626 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:11:56.005709 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.005717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.005951 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.006124 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:56.006483 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.007741 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.009360 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.010210 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.010977 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.011131 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.011190 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.011296 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":111,"threads_started":1}
I20260924 20:11:56.011555 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.005594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.0.115.129:39269)
0924 20:11:56.005904 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.005918 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.005936 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:11:56.006000 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.006004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.006159 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.006638 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:56.006793 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.007933 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.009362 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.010210 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.010438 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.010611 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.011310 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.011406 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260924 20:11:56.013891 17893 tablet_service.cc:1467] Processing CreateTablet for tablet 8953d6893092419cbb5c92ea0a8ef8c8 (DEFAULT_TABLE table=test-table [id=503fe79abc8c4d97be775e4254cf55e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:56.014292 17625 tablet_service.cc:1467] Processing CreateTablet for tablet 8953d6893092419cbb5c92ea0a8ef8c8 (DEFAULT_TABLE table=test-table [id=503fe79abc8c4d97be775e4254cf55e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:56.014292 17759 tablet_service.cc:1467] Processing CreateTablet for tablet 8953d6893092419cbb5c92ea0a8ef8c8 (DEFAULT_TABLE table=test-table [id=503fe79abc8c4d97be775e4254cf55e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:56.015033 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8953d6893092419cbb5c92ea0a8ef8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:56.015323 17625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8953d6893092419cbb5c92ea0a8ef8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:56.015327 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8953d6893092419cbb5c92ea0a8ef8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:56.021584 17976 tablet_bootstrap.cc:492] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Bootstrap starting.
I20260924 20:11:56.022910 17978 tablet_bootstrap.cc:492] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Bootstrap starting.
I20260924 20:11:56.023200 17977 tablet_bootstrap.cc:492] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5: Bootstrap starting.
I20260924 20:11:56.024256 17976 tablet_bootstrap.cc:654] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:56.025197 17977 tablet_bootstrap.cc:654] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:56.025633 17978 tablet_bootstrap.cc:654] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:56.026013 17976 log.cc:824] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:56.026206 17977 log.cc:824] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:56.026572 17978 log.cc:824] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:56.028164 17977 tablet_bootstrap.cc:492] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5: No bootstrap required, opened a new log
I20260924 20:11:56.028249 17976 tablet_bootstrap.cc:492] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: No bootstrap required, opened a new log
I20260924 20:11:56.028393 17977 ts_tablet_manager.cc:1397] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:11:56.028456 17976 ts_tablet_manager.cc:1397] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:11:56.028523 17978 tablet_bootstrap.cc:492] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: No bootstrap required, opened a new log
I20260924 20:11:56.028676 17978 ts_tablet_manager.cc:1397] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:11:56.031773 17977 raft_consensus.cc:348] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.032094 17977 raft_consensus.cc:374] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:56.032153 17977 raft_consensus.cc:729] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af77514638544809817d7f3e23bd59c5, State: Initialized, Role: FOLLOWER
I20260924 20:11:56.032382 17976 raft_consensus.cc:348] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.032382 17978 raft_consensus.cc:348] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.032631 17978 raft_consensus.cc:374] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:56.032631 17976 raft_consensus.cc:374] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:56.032680 17978 raft_consensus.cc:729] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c21e5c87ef845f8af4f86d7e2b3653c, State: Initialized, Role: FOLLOWER
I20260924 20:11:56.032691 17976 raft_consensus.cc:729] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059370ae5ee94117ba5e26f22623e3e6, State: Initialized, Role: FOLLOWER
I20260924 20:11:56.032657 17977 consensus_queue.cc:260] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.033131 17978 consensus_queue.cc:260] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.033131 17976 consensus_queue.cc:260] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.033412 17691 heartbeater.cc:503] Master 127.0.115.190:41271 was elected leader, sending a full tablet report...
I20260924 20:11:56.033764 17977 ts_tablet_manager.cc:1428] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:11:56.034102 17978 ts_tablet_manager.cc:1428] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:11:56.034453 17976 ts_tablet_manager.cc:1428] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:11:56.034561 17825 heartbeater.cc:503] Master 127.0.115.190:41271 was elected leader, sending a full tablet report...
I20260924 20:11:56.034718 17959 heartbeater.cc:503] Master 127.0.115.190:41271 was elected leader, sending a full tablet report...
I20260924 20:11:56.079174 17982 raft_consensus.cc:482] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:56.079545 17982 raft_consensus.cc:504] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.081081 17982 leader_election.cc:288] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059370ae5ee94117ba5e26f22623e3e6 (127.0.115.131:36775), 1c21e5c87ef845f8af4f86d7e2b3653c (127.0.115.130:43787)
I20260924 20:11:56.086530 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.081479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:43619 (local address 127.0.115.131:36775)
0924 20:11:56.081603 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.081607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.081786 (+ 179us) server_negotiation.cc:407] Connection header received
0924 20:11:56.081927 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.081930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.081981 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.082067 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:56.082509 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.083329 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.085248 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.085893 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.086019 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.086074 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.086321 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.086396 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:11:56.086530 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.081448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:39227 (local address 127.0.115.130:43787)
0924 20:11:56.081576 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.081581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.081785 (+ 204us) server_negotiation.cc:407] Connection header received
0924 20:11:56.081927 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.081930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.081984 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.082069 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:56.082509 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.083349 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.085248 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.085883 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.086019 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.086074 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.086324 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.086396 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:11:56.086584 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.081260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:36775 (local address 127.0.115.129:43619)
0924 20:11:56.081665 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.081681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:56.081819 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:56.082093 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:56.082096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:56.082122 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:56.082356 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.082364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.083457 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.083460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.085102 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.085115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.086020 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.086024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.086181 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.086217 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 20:11:56.086286 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:11:56.086390 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":311,"spinlock_wait_cycles":23936,"thread_start_us":140,"threads_started":1}
I20260924 20:11:56.086589 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.081273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:43787 (local address 127.0.115.129:39227)
0924 20:11:56.081691 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.081706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:56.081819 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:56.082093 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:56.082096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:56.082122 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:56.082356 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.082364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.083464 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.083466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.085102 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.085115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.086020 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.086024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.086164 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.086207 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 20:11:56.086286 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:11:56.086390 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":198,"spinlock_wait_cycles":30080,"thread_start_us":116,"threads_started":1}
I20260924 20:11:56.087312 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953d6893092419cbb5c92ea0a8ef8c8" candidate_uuid: "af77514638544809817d7f3e23bd59c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" is_pre_election: true
I20260924 20:11:56.087656 17779 raft_consensus.cc:2454] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af77514638544809817d7f3e23bd59c5 in term 0.
I20260924 20:11:56.087729 17913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953d6893092419cbb5c92ea0a8ef8c8" candidate_uuid: "af77514638544809817d7f3e23bd59c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059370ae5ee94117ba5e26f22623e3e6" is_pre_election: true
I20260924 20:11:56.088053 17913 raft_consensus.cc:2454] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af77514638544809817d7f3e23bd59c5 in term 0.
I20260924 20:11:56.088191 17580 leader_election.cc:302] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c21e5c87ef845f8af4f86d7e2b3653c, af77514638544809817d7f3e23bd59c5; no voters:
I20260924 20:11:56.088511 17982 raft_consensus.cc:2781] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:11:56.088598 17982 raft_consensus.cc:482] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:11:56.088629 17982 raft_consensus.cc:3037] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:56.089696 17982 raft_consensus.cc:504] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.090081 17982 leader_election.cc:288] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [CANDIDATE]: Term 1 election: Requested vote from peers 059370ae5ee94117ba5e26f22623e3e6 (127.0.115.131:36775), 1c21e5c87ef845f8af4f86d7e2b3653c (127.0.115.130:43787)
I20260924 20:11:56.090458 17913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953d6893092419cbb5c92ea0a8ef8c8" candidate_uuid: "af77514638544809817d7f3e23bd59c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059370ae5ee94117ba5e26f22623e3e6"
I20260924 20:11:56.090462 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953d6893092419cbb5c92ea0a8ef8c8" candidate_uuid: "af77514638544809817d7f3e23bd59c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c"
I20260924 20:11:56.090606 17779 raft_consensus.cc:3037] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:56.090603 17913 raft_consensus.cc:3037] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:56.092046 17779 raft_consensus.cc:2454] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af77514638544809817d7f3e23bd59c5 in term 1.
I20260924 20:11:56.092046 17913 raft_consensus.cc:2454] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af77514638544809817d7f3e23bd59c5 in term 1.
I20260924 20:11:56.092445 17581 leader_election.cc:302] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 059370ae5ee94117ba5e26f22623e3e6, 1c21e5c87ef845f8af4f86d7e2b3653c, af77514638544809817d7f3e23bd59c5; no voters:
I20260924 20:11:56.092646 17982 raft_consensus.cc:2781] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:56.092896 17982 raft_consensus.cc:686] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 1 LEADER]: Becoming Leader. State: Replica: af77514638544809817d7f3e23bd59c5, State: Running, Role: LEADER
I20260924 20:11:56.093286 17982 consensus_queue.cc:237] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } }
I20260924 20:11:56.096729 17500 catalog_manager.cc:5495] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to af77514638544809817d7f3e23bd59c5 (127.0.115.129). New cstate: current_term: 1 leader_uuid: "af77514638544809817d7f3e23bd59c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af77514638544809817d7f3e23bd59c5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 39269 } health_report { overall_health: HEALTHY } } }
I20260924 20:11:56.107929 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.104604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.0.115.129:39269)
0924 20:11:56.104727 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.104730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.104743 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:11:56.104826 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.104829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.104886 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.104977 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:11:56.105191 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.105688 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.106464 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.106668 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.106742 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.106989 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:11:56.107427 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:11:56.107519 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.107664 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.107748 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:11:56.113788 17913 raft_consensus.cc:1261] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 1 FOLLOWER]: Refusing update from remote peer af77514638544809817d7f3e23bd59c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:56.113890 17779 raft_consensus.cc:1261] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 1 FOLLOWER]: Refusing update from remote peer af77514638544809817d7f3e23bd59c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:56.114567 17982 consensus_queue.cc:1035] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059370ae5ee94117ba5e26f22623e3e6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 36775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:56.114789 17987 consensus_queue.cc:1035] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c21e5c87ef845f8af4f86d7e2b3653c" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:56.122805 17991 mvcc.cc:205] Tried to move back new op lower bound from 7332989813194764288 to 7332989813121056768. Current Snapshot: MvccSnapshot[applied={T|T < 7332989813194764288}]
I20260924 20:11:56.124673 17992 mvcc.cc:205] Tried to move back new op lower bound from 7332989813194764288 to 7332989813121056768. Current Snapshot: MvccSnapshot[applied={T|T < 7332989813194764288}]
I20260924 20:11:56.132380 17500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:56.132568 17500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:56.135048 17500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d865561f56ee4344b04c379098970e40: Sending DeleteTablet for 3 replicas of tablet 8953d6893092419cbb5c92ea0a8ef8c8
I20260924 20:11:56.135819 17625 tablet_service.cc:1514] Processing DeleteTablet for tablet 8953d6893092419cbb5c92ea0a8ef8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:56 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:36322
I20260924 20:11:56.135821 17759 tablet_service.cc:1514] Processing DeleteTablet for tablet 8953d6893092419cbb5c92ea0a8ef8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:56 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:34938
I20260924 20:11:56.135860 17893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8953d6893092419cbb5c92ea0a8ef8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:56 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:59148
I20260924 20:11:56.136404 18001 tablet_replica.cc:330] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: stopping tablet replica
I20260924 20:11:56.136456 18002 tablet_replica.cc:330] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5: stopping tablet replica
I20260924 20:11:56.136739 18001 raft_consensus.cc:2229] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:11:56.136775 18002 raft_consensus.cc:2229] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:11:56.137108 18001 raft_consensus.cc:2258] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:56.137210 18002 raft_consensus.cc:2258] T 8953d6893092419cbb5c92ea0a8ef8c8 P af77514638544809817d7f3e23bd59c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:56.137792 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 17562
I20260924 20:11:56.138492 18000 tablet_replica.cc:330] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: stopping tablet replica
I20260924 20:11:56.138756 18000 raft_consensus.cc:2229] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:11:56.138862 18001 ts_tablet_manager.cc:1905] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:56.139024 18000 raft_consensus.cc:2258] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:56.140573 18000 ts_tablet_manager.cc:1905] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:56.141974 18001 ts_tablet_manager.cc:1918] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:56.142093 18001 log.cc:1196] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/8953d6893092419cbb5c92ea0a8ef8c8
I20260924 20:11:56.142396 18001 ts_tablet_manager.cc:1939] T 8953d6893092419cbb5c92ea0a8ef8c8 P 059370ae5ee94117ba5e26f22623e3e6: Deleting consensus metadata
I20260924 20:11:56.143517 17487 catalog_manager.cc:4841] TS 059370ae5ee94117ba5e26f22623e3e6 (127.0.115.131:36775): tablet 8953d6893092419cbb5c92ea0a8ef8c8 (table test-table [id=503fe79abc8c4d97be775e4254cf55e3]) successfully deleted
I20260924 20:11:56.143958 18000 ts_tablet_manager.cc:1918] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:56.144061 18000 log.cc:1196] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/8953d6893092419cbb5c92ea0a8ef8c8
I20260924 20:11:56.144403 18000 ts_tablet_manager.cc:1939] T 8953d6893092419cbb5c92ea0a8ef8c8 P 1c21e5c87ef845f8af4f86d7e2b3653c: Deleting consensus metadata
I20260924 20:11:56.145642 17486 catalog_manager.cc:4841] TS 1c21e5c87ef845f8af4f86d7e2b3653c (127.0.115.130:43787): tablet 8953d6893092419cbb5c92ea0a8ef8c8 (table test-table [id=503fe79abc8c4d97be775e4254cf55e3]) successfully deleted
W20260924 20:11:56.147007 17484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:39269 (error 108)
I20260924 20:11:56.147724 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 17696
I20260924 20:11:56.148960 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.147739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:39269 (local address 127.0.0.1:36336)
0924 20:11:56.147903 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.148850 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 20:11:56.149140 17484 catalog_manager.cc:4584] TS af77514638544809817d7f3e23bd59c5 (127.0.115.129:39269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8953d6893092419cbb5c92ea0a8ef8c8: Network error: Client connection negotiation failed: client connection to 127.0.115.129:39269: connect: Connection refused (error 111)
I20260924 20:11:56.154633 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 17830
I20260924 20:11:56.160377 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 17468
2026-09-24T20:11:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2310 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T20:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:11:56Z Disabled control of system clock
I20260924 20:11:56.191221 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34859
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:11:56.302373 18007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:56.302630 18007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:56.302664 18007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:56.306692 18007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:11:56.306752 18007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:56.306771 18007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:56.306788 18007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:11:56.306803 18007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:11:56.311594 18007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:34859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:56.312785 18007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:56.314152 18007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:56.323304 18015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.323457 18013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.323304 18012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:56.323628 18007 server_base.cc:1034] running on GCE node
I20260924 20:11:56.324327 18007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:56.325508 18007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:56.326706 18007 hybrid_clock.cc:648] HybridClock initialized: now 1790280716326677 us; error 41 us; skew 500 ppm
I20260924 20:11:56.328677 18007 webserver.cc:460] Webserver started at http://127.0.115.190:42217/ using document root <none> and password file <none>
I20260924 20:11:56.329264 18007 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:56.329365 18007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:56.329598 18007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:56.331676 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "384d721ee0d342a6bdca99c754a000d1"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "ec9a0eb51e992942df2a1feeb258b467"
server_key_iv: "516386a26e3c4843dbf3bf6308986d55"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.332288 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "384d721ee0d342a6bdca99c754a000d1"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "ec9a0eb51e992942df2a1feeb258b467"
server_key_iv: "516386a26e3c4843dbf3bf6308986d55"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.335928 18007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:11:56.338423 18021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.339684 18007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:11:56.339850 18007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "384d721ee0d342a6bdca99c754a000d1"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "ec9a0eb51e992942df2a1feeb258b467"
server_key_iv: "516386a26e3c4843dbf3bf6308986d55"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.340003 18007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:56.354270 18007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:56.355013 18007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:56.355221 18007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:56.362193 18007 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:34859
I20260924 20:11:56.362205 18073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:34859 every 8 connection(s)
I20260924 20:11:56.363200 18007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:11:56.366333 18074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:56.366878 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18007
I20260924 20:11:56.367029 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:11:56.367317 462 external_mini_cluster.cc:1428] Setting key c6b0249f34b30368f50035c498729e4d
I20260924 20:11:56.373392 18074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1: Bootstrap starting.
I20260924 20:11:56.376273 18074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:56.377246 18074 log.cc:824] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:56.378422 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.368747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.0.115.190:34859)
0924 20:11:56.369485 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.369505 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.369549 (+ 44us) server_negotiation.cc:407] Connection header received
0924 20:11:56.370394 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.370416 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.370934 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.371312 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:56.371787 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.373288 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.374341 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.375061 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.375218 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.376289 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:56.376298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:56.376316 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:11:56.376868 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:56.376937 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:11:56.376964 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:56.377067 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.377236 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.377530 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260924 20:11:56.379577 18074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1: No bootstrap required, opened a new log
I20260924 20:11:56.383558 18074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384d721ee0d342a6bdca99c754a000d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34859 } }
I20260924 20:11:56.383800 18074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:56.383836 18074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384d721ee0d342a6bdca99c754a000d1, State: Initialized, Role: FOLLOWER
I20260924 20:11:56.384315 18074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384d721ee0d342a6bdca99c754a000d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34859 } }
I20260924 20:11:56.384442 18074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:11:56.384527 18074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:11:56.384649 18074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:56.385625 18074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384d721ee0d342a6bdca99c754a000d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34859 } }
I20260924 20:11:56.385985 18074 leader_election.cc:302] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384d721ee0d342a6bdca99c754a000d1; no voters:
I20260924 20:11:56.386296 18074 leader_election.cc:288] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:11:56.386443 18079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:56.386694 18079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [term 1 LEADER]: Becoming Leader. State: Replica: 384d721ee0d342a6bdca99c754a000d1, State: Running, Role: LEADER
I20260924 20:11:56.387033 18079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384d721ee0d342a6bdca99c754a000d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34859 } }
I20260924 20:11:56.387554 18074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:11:56.388759 18081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384d721ee0d342a6bdca99c754a000d1. Latest consensus state: current_term: 1 leader_uuid: "384d721ee0d342a6bdca99c754a000d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384d721ee0d342a6bdca99c754a000d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34859 } } }
I20260924 20:11:56.388895 18081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:56.389212 18080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384d721ee0d342a6bdca99c754a000d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384d721ee0d342a6bdca99c754a000d1" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 34859 } } }
I20260924 20:11:56.389317 18080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1 [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:56.390125 18088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:11:56.392917 18088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:11:56.398656 18088 catalog_manager.cc:1347] Generated new cluster ID: 2b36494048e8433aa506df912ac3b7a1
I20260924 20:11:56.398736 18088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:11:56.418100 18088 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:11:56.419077 18088 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:11:56.427214 18088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1: Generated new TSK 0
I20260924 20:11:56.427996 18088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:11:56.436655 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34859
--builtin_ntp_servers=127.0.115.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:11:56.548717 18098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:56.549041 18098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:56.549127 18098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:56.553197 18098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:56.553275 18098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:56.553369 18098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:11:56.558156 18098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:56.559299 18098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:56.560804 18098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:56.571043 18104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.571085 18103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.571041 18106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:56.571259 18098 server_base.cc:1034] running on GCE node
I20260924 20:11:56.571792 18098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:56.572520 18098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:56.575807 18098 hybrid_clock.cc:648] HybridClock initialized: now 1790280716575789 us; error 28 us; skew 500 ppm
I20260924 20:11:56.577795 18098 webserver.cc:460] Webserver started at http://127.0.115.129:35385/ using document root <none> and password file <none>
I20260924 20:11:56.578408 18098 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:56.578510 18098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:56.578745 18098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:56.580915 18098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "10c1b158c6cb4374883a5a553d9bc4d5"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "3309cd1dc4483d1d26b04be7d36ad71a"
server_key_iv: "20dc13ba479c1035a44031e246238d3f"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.581493 18098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "10c1b158c6cb4374883a5a553d9bc4d5"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "3309cd1dc4483d1d26b04be7d36ad71a"
server_key_iv: "20dc13ba479c1035a44031e246238d3f"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.585353 18098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:11:56.587791 18112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.589332 18098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:11:56.589426 18098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "10c1b158c6cb4374883a5a553d9bc4d5"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "3309cd1dc4483d1d26b04be7d36ad71a"
server_key_iv: "20dc13ba479c1035a44031e246238d3f"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.589591 18098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:56.610354 18098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:56.611073 18098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:56.611281 18098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:56.611902 18098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:56.612932 18098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:56.612980 18098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.613049 18098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:56.613081 18098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.622750 18098 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:36989
I20260924 20:11:56.622777 18225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:36989 every 8 connection(s)
I20260924 20:11:56.623672 18098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:11:56.633481 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.625616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:43593 (local address 127.0.115.190:34859)
0924 20:11:56.625765 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.625769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.626924 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 20:11:56.628548 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.628551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.628597 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.628671 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:56.629672 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.630200 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.631458 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.631707 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.631807 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.633059 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:56.633062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:56.633064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:11:56.633076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:56.633095 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:11:56.633103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:56.633163 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.633313 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.633365 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:11:56.633953 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18098
I20260924 20:11:56.634096 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:11:56.634414 462 external_mini_cluster.cc:1428] Setting key 1923e737ee6217370c9a61cdf940fd30
I20260924 20:11:56.634599 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.626020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:34859 (local address 127.0.115.129:43593)
0924 20:11:56.626765 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.626816 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:56.628345 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:56.628857 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:56.628865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:56.628955 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:56.629539 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.629555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.630374 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.630377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.631348 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.631358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.631866 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.631870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.632118 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.632802 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:11:56.632803 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:56.632844 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:11:56.632908 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:56.633180 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:56.633187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:56.633192 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:11:56.633300 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:11:56.633520 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":112,"threads_started":1}
I20260924 20:11:56.636014 18226 heartbeater.cc:347] Connected to a master server at 127.0.115.190:34859
I20260924 20:11:56.636277 18226 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:56.636785 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34859
--builtin_ntp_servers=127.0.115.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:11:56.636881 18226 heartbeater.cc:511] Master 127.0.115.190:34859 requested a full tablet report, sending...
I20260924 20:11:56.638571 18038 ts_manager.cc:194] Registered new tserver with Master: 10c1b158c6cb4374883a5a553d9bc4d5 (127.0.115.129:36989)
I20260924 20:11:56.639812 18038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.129:43593
W20260924 20:11:56.747817 18229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:56.748168 18229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:56.748253 18229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:56.752218 18229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:56.752307 18229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:56.752398 18229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:11:56.757231 18229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:56.758350 18229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:56.759752 18229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:56.769654 18235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.769654 18234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.769721 18237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:56.769891 18229 server_base.cc:1034] running on GCE node
I20260924 20:11:56.770409 18229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:56.771173 18229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:56.772425 18229 hybrid_clock.cc:648] HybridClock initialized: now 1790280716772385 us; error 59 us; skew 500 ppm
I20260924 20:11:56.774401 18229 webserver.cc:460] Webserver started at http://127.0.115.130:43619/ using document root <none> and password file <none>
I20260924 20:11:56.775010 18229 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:56.775104 18229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:56.775348 18229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:56.777500 18229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "21b34719283646b69725a07759383f2d"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "13fab5c378fcf3911e075ff8381264c0"
server_key_iv: "32737342c576fce17318839cbd2af62d"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.778103 18229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "21b34719283646b69725a07759383f2d"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "13fab5c378fcf3911e075ff8381264c0"
server_key_iv: "32737342c576fce17318839cbd2af62d"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.781862 18229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:11:56.784332 18243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.785578 18229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:11:56.785722 18229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "21b34719283646b69725a07759383f2d"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "13fab5c378fcf3911e075ff8381264c0"
server_key_iv: "32737342c576fce17318839cbd2af62d"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.785837 18229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:56.805176 18229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:56.805934 18229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:56.806141 18229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:56.806823 18229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:56.808108 18229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:56.808171 18229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.808238 18229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:56.808295 18229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.819286 18229 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:44035
I20260924 20:11:56.819304 18356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:44035 every 8 connection(s)
I20260924 20:11:56.820339 18229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:11:56.823437 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18229
I20260924 20:11:56.823568 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:11:56.823877 462 external_mini_cluster.cc:1428] Setting key 39d09fe952d6d9bb342d75d212384eea
I20260924 20:11:56.825918 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34859
--builtin_ntp_servers=127.0.115.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:11:56.830400 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.822268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:54543 (local address 127.0.115.190:34859)
0924 20:11:56.822412 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:56.822416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:56.823475 (+ 1059us) server_negotiation.cc:407] Connection header received
0924 20:11:56.825165 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:56.825169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:56.825222 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:56.825300 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:56.826271 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.826798 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.828436 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.828665 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.828772 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.829894 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:56.829898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:56.829901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:11:56.829914 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:56.829940 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:11:56.829949 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:56.830034 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:56.830198 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 20:11:56.830260 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:11:56.831262 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:56.822648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:34859 (local address 127.0.115.130:54543)
0924 20:11:56.823324 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:56.823364 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:56.824955 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:56.825476 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:56.825483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:56.825569 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:56.826135 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.826151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.826926 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.826930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.828311 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:56.828322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:56.828824 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:56.828828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:56.829007 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:56.829637 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:11:56.829638 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:56.829679 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:11:56.829742 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:56.830035 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:56.830047 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:56.830055 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:11:56.830186 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 20:11:56.830484 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260924 20:11:56.832450 18357 heartbeater.cc:347] Connected to a master server at 127.0.115.190:34859
I20260924 20:11:56.832716 18357 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:56.833227 18357 heartbeater.cc:511] Master 127.0.115.190:34859 requested a full tablet report, sending...
I20260924 20:11:56.834335 18038 ts_manager.cc:194] Registered new tserver with Master: 21b34719283646b69725a07759383f2d (127.0.115.130:44035)
I20260924 20:11:56.835222 18038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.130:54543
W20260924 20:11:56.944816 18360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:56.945111 18360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:56.945155 18360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:56.949785 18360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:56.949853 18360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:56.949918 18360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:11:56.955339 18360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:56.956525 18360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:56.957994 18360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:56.968713 18368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.968713 18366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:56.968766 18365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:56.969058 18360 server_base.cc:1034] running on GCE node
I20260924 20:11:56.969571 18360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:56.970266 18360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:56.971455 18360 hybrid_clock.cc:648] HybridClock initialized: now 1790280716971419 us; error 49 us; skew 500 ppm
I20260924 20:11:56.973426 18360 webserver.cc:460] Webserver started at http://127.0.115.131:44531/ using document root <none> and password file <none>
I20260924 20:11:56.974056 18360 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:56.974121 18360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:56.974359 18360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:56.976608 18360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "6bda497bfff04b94b7e700298ed2bb5e"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "89e1bd01d52cb6531c5344d3740fc9f5"
server_key_iv: "67022a3b77d9ce0da8670fb0e538665d"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.977186 18360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "6bda497bfff04b94b7e700298ed2bb5e"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "89e1bd01d52cb6531c5344d3740fc9f5"
server_key_iv: "67022a3b77d9ce0da8670fb0e538665d"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.981040 18360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:11:56.983484 18374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:56.984798 18360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:11:56.984935 18360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "6bda497bfff04b94b7e700298ed2bb5e"
format_stamp: "Formatted at 2026-09-24 20:11:56 on dist-test-slave-2pm7"
server_key: "89e1bd01d52cb6531c5344d3740fc9f5"
server_key_iv: "67022a3b77d9ce0da8670fb0e538665d"
server_key_version: "encryptionkey@0"
I20260924 20:11:56.985057 18360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:56.997382 18360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:56.998083 18360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:56.998282 18360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:56.998878 18360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:56.999876 18360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:56.999949 18360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:57.000068 18360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:57.000119 18360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:57.008940 18360 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:46545
I20260924 20:11:57.008967 18487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:46545 every 8 connection(s)
I20260924 20:11:57.009826 18360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:11:57.012913 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18360
I20260924 20:11:57.013023 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:11:57.013367 462 external_mini_cluster.cc:1428] Setting key a3cb972bff069c7936796ef95e25e3df
I20260924 20:11:57.020099 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.011737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:55579 (local address 127.0.115.190:34859)
0924 20:11:57.011916 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.011921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.013180 (+ 1259us) server_negotiation.cc:407] Connection header received
0924 20:11:57.015105 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.015108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.015158 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.015243 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:57.016376 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.016900 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.018092 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.018303 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.018377 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.019636 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:57.019638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:57.019640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:11:57.019651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:57.019667 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:11:57.019674 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:57.019727 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.019875 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.019925 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 20:11:57.020974 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.012209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:34859 (local address 127.0.115.131:55579)
0924 20:11:57.013009 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:57.013066 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:57.014899 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:57.015502 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:57.015513 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:57.015643 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:57.016244 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.016267 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.017024 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.017026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.017989 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.017999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.018521 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.018528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.018745 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.019399 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:11:57.019400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:57.019440 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:11:57.019506 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:57.019745 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:57.019752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:57.019756 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:11:57.019862 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 20:11:57.020146 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":104,"threads_started":1}
I20260924 20:11:57.022228 18488 heartbeater.cc:347] Connected to a master server at 127.0.115.190:34859
I20260924 20:11:57.022522 18488 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:57.023052 18488 heartbeater.cc:511] Master 127.0.115.190:34859 requested a full tablet report, sending...
I20260924 20:11:57.024127 18038 ts_manager.cc:194] Registered new tserver with Master: 6bda497bfff04b94b7e700298ed2bb5e (127.0.115.131:46545)
I20260924 20:11:57.024690 18038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.131:55579
I20260924 20:11:57.027482 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:11:57.033389 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.029431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.115.190:34859)
0924 20:11:57.029632 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.029637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.029785 (+ 148us) server_negotiation.cc:407] Connection header received
0924 20:11:57.029952 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.029955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.030000 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.030094 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:57.030442 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.030932 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.031779 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.032101 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.032217 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.032926 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:57.032930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:57.032933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:11:57.032945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:57.032965 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:11:57.032982 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:57.033064 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.033177 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.033232 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 20:11:57.036213 18038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:11:57.039122 18038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:11:57.054107 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.047911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:44035 (local address 127.0.0.1:50184)
0924 20:11:57.048293 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:57.048347 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:57.048479 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:57.049108 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:57.049117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:57.049150 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:57.049398 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.049410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.051074 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.051078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.052479 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.052489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.053561 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.053565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.053698 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.053737 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 20:11:57.053833 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 20:11:57.053926 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":22,"thread_start_us":88,"threads_started":1}
I20260924 20:11:57.054157 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.048062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.0.115.130:44035)
0924 20:11:57.048433 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.048445 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.048472 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:11:57.048665 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.048673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.048912 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.049574 (+ 662us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:57.049728 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.050951 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.052586 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.053417 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.053657 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.053823 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.053874 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.053975 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260924 20:11:57.054373 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.048446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.0.115.129:36989)
0924 20:11:57.048798 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.048812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.049185 (+ 373us) server_negotiation.cc:407] Connection header received
0924 20:11:57.049399 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.049404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.049607 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.049765 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:57.050198 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.051286 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.052653 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.053417 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.053536 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.053754 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.054070 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.054183 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":143,"threads_started":1}
I20260924 20:11:57.054672 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.047911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:36989 (local address 127.0.0.1:49254)
0924 20:11:57.049071 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:57.049085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:57.049265 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:57.049921 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:57.049925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:57.049943 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:57.050056 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.050065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.051409 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.051412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.052515 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.052525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.053821 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.053825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.053913 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.053934 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:11:57.054014 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:11:57.054093 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":940,"thread_start_us":106,"threads_started":1}
I20260924 20:11:57.055547 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.049216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:46545 (local address 127.0.0.1:54266)
0924 20:11:57.049825 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:57.049845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:57.049928 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:57.050470 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:57.050473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:57.050490 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:57.050571 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.050577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.051760 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.051763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.052782 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.052789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.053545 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.053549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.055249 (+ 1700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.055273 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:11:57.055354 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:11:57.055421 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260924 20:11:57.056088 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.049631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.0.115.131:46545)
0924 20:11:57.050053 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.050070 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.050093 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:11:57.050171 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.050175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.050330 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.050491 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:57.050663 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.051635 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.052892 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.053414 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.053543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.053743 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.055810 (+ 2067us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.055911 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":130,"threads_started":1}
I20260924 20:11:57.057466 18291 tablet_service.cc:1467] Processing CreateTablet for tablet a9d56dcc3d5047de9dc310deb393133d (DEFAULT_TABLE table=test-table [id=42338b406f0040269ee31b54c4a02d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:57.057500 18160 tablet_service.cc:1467] Processing CreateTablet for tablet a9d56dcc3d5047de9dc310deb393133d (DEFAULT_TABLE table=test-table [id=42338b406f0040269ee31b54c4a02d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:57.058454 18291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d56dcc3d5047de9dc310deb393133d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:57.058552 18160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d56dcc3d5047de9dc310deb393133d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:57.059051 18422 tablet_service.cc:1467] Processing CreateTablet for tablet a9d56dcc3d5047de9dc310deb393133d (DEFAULT_TABLE table=test-table [id=42338b406f0040269ee31b54c4a02d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:57.060042 18422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d56dcc3d5047de9dc310deb393133d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:57.064525 18505 tablet_bootstrap.cc:492] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Bootstrap starting.
I20260924 20:11:57.066280 18506 tablet_bootstrap.cc:492] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5: Bootstrap starting.
I20260924 20:11:57.066782 18505 tablet_bootstrap.cc:654] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:57.067507 18507 tablet_bootstrap.cc:492] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Bootstrap starting.
I20260924 20:11:57.067636 18505 log.cc:824] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:57.068463 18506 tablet_bootstrap.cc:654] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:57.069380 18506 log.cc:824] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:57.069717 18507 tablet_bootstrap.cc:654] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:57.070220 18505 tablet_bootstrap.cc:492] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: No bootstrap required, opened a new log
I20260924 20:11:57.070420 18505 ts_tablet_manager.cc:1397] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:11:57.070529 18507 log.cc:824] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:57.071396 18506 tablet_bootstrap.cc:492] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5: No bootstrap required, opened a new log
I20260924 20:11:57.071615 18506 ts_tablet_manager.cc:1397] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:11:57.072657 18507 tablet_bootstrap.cc:492] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: No bootstrap required, opened a new log
I20260924 20:11:57.072835 18507 ts_tablet_manager.cc:1397] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:11:57.074829 18505 raft_consensus.cc:348] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.075210 18505 raft_consensus.cc:374] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:57.075281 18505 raft_consensus.cc:729] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b34719283646b69725a07759383f2d, State: Initialized, Role: FOLLOWER
I20260924 20:11:57.075155 18506 raft_consensus.cc:348] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.075433 18506 raft_consensus.cc:374] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:57.075549 18506 raft_consensus.cc:729] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c1b158c6cb4374883a5a553d9bc4d5, State: Initialized, Role: FOLLOWER
I20260924 20:11:57.075752 18505 consensus_queue.cc:260] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.076043 18506 consensus_queue.cc:260] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.076475 18357 heartbeater.cc:503] Master 127.0.115.190:34859 was elected leader, sending a full tablet report...
I20260924 20:11:57.077046 18506 ts_tablet_manager.cc:1428] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:11:57.076893 18507 raft_consensus.cc:348] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.077128 18507 raft_consensus.cc:374] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:57.077173 18507 raft_consensus.cc:729] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bda497bfff04b94b7e700298ed2bb5e, State: Initialized, Role: FOLLOWER
I20260924 20:11:57.077543 18226 heartbeater.cc:503] Master 127.0.115.190:34859 was elected leader, sending a full tablet report...
I20260924 20:11:57.077540 18507 consensus_queue.cc:260] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.078148 18488 heartbeater.cc:503] Master 127.0.115.190:34859 was elected leader, sending a full tablet report...
I20260924 20:11:57.078439 18507 ts_tablet_manager.cc:1428] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:11:57.078800 18505 ts_tablet_manager.cc:1428] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 20:11:57.122112 18513 raft_consensus.cc:482] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:57.122365 18513 raft_consensus.cc:504] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.123641 18513 leader_election.cc:288] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b34719283646b69725a07759383f2d (127.0.115.130:44035), 10c1b158c6cb4374883a5a553d9bc4d5 (127.0.115.129:36989)
W20260924 20:11:57.124869 18227 tablet.cc:2369] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:11:57.127842 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.123830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:44035 (local address 127.0.115.131:59369)
0924 20:11:57.123982 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:57.124000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:57.124104 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:57.124560 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:57.124564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:57.124583 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:57.124704 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.124710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.125799 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.125802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.126648 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.126657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.127365 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.127369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.127493 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.127515 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:11:57.127589 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:11:57.127661 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":26}
I20260924 20:11:57.127856 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.124089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:59369 (local address 127.0.115.130:44035)
0924 20:11:57.124291 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.124295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.124355 (+ 60us) server_negotiation.cc:407] Connection header received
0924 20:11:57.124406 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.124408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.124451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.124547 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:57.124882 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.125686 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.126787 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.127237 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.127343 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.127384 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.127629 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.127722 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 20:11:57.127923 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.124089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:37095 (local address 127.0.115.129:36989)
0924 20:11:57.124222 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.124227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.124418 (+ 191us) server_negotiation.cc:407] Connection header received
0924 20:11:57.124584 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.124587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.124640 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.124755 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:57.125062 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.125792 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.126934 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.127364 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.127483 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.127529 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.127729 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.127804 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:11:57.128022 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.123981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:36989 (local address 127.0.115.131:37095)
0924 20:11:57.124325 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:57.124336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:57.124469 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:57.124785 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:57.124791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:57.124812 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:57.124928 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.124935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.125930 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.125933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.126818 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:57.126828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.127487 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.127490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:57.127596 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.127623 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:11:57.127700 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:11:57.127758 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":13,"thread_start_us":127,"threads_started":1}
I20260924 20:11:57.128788 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d56dcc3d5047de9dc310deb393133d" candidate_uuid: "6bda497bfff04b94b7e700298ed2bb5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" is_pre_election: true
I20260924 20:11:57.128787 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d56dcc3d5047de9dc310deb393133d" candidate_uuid: "6bda497bfff04b94b7e700298ed2bb5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b34719283646b69725a07759383f2d" is_pre_election: true
I20260924 20:11:57.129120 18311 raft_consensus.cc:2454] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bda497bfff04b94b7e700298ed2bb5e in term 0.
I20260924 20:11:57.129120 18180 raft_consensus.cc:2454] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bda497bfff04b94b7e700298ed2bb5e in term 0.
I20260924 20:11:57.129729 18376 leader_election.cc:302] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10c1b158c6cb4374883a5a553d9bc4d5, 21b34719283646b69725a07759383f2d, 6bda497bfff04b94b7e700298ed2bb5e; no voters:
I20260924 20:11:57.130043 18513 raft_consensus.cc:2781] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:11:57.130137 18513 raft_consensus.cc:482] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:11:57.130198 18513 raft_consensus.cc:3037] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:57.131177 18513 raft_consensus.cc:504] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.131572 18513 leader_election.cc:288] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [CANDIDATE]: Term 1 election: Requested vote from peers 21b34719283646b69725a07759383f2d (127.0.115.130:44035), 10c1b158c6cb4374883a5a553d9bc4d5 (127.0.115.129:36989)
I20260924 20:11:57.131889 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d56dcc3d5047de9dc310deb393133d" candidate_uuid: "6bda497bfff04b94b7e700298ed2bb5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b34719283646b69725a07759383f2d"
I20260924 20:11:57.131928 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d56dcc3d5047de9dc310deb393133d" candidate_uuid: "6bda497bfff04b94b7e700298ed2bb5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c1b158c6cb4374883a5a553d9bc4d5"
I20260924 20:11:57.132061 18180 raft_consensus.cc:3037] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:57.132050 18311 raft_consensus.cc:3037] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:57.133347 18180 raft_consensus.cc:2454] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bda497bfff04b94b7e700298ed2bb5e in term 1.
I20260924 20:11:57.133344 18311 raft_consensus.cc:2454] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bda497bfff04b94b7e700298ed2bb5e in term 1.
I20260924 20:11:57.133664 18378 leader_election.cc:302] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b34719283646b69725a07759383f2d, 6bda497bfff04b94b7e700298ed2bb5e; no voters:
I20260924 20:11:57.133848 18513 raft_consensus.cc:2781] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:57.134047 18513 raft_consensus.cc:686] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 1 LEADER]: Becoming Leader. State: Replica: 6bda497bfff04b94b7e700298ed2bb5e, State: Running, Role: LEADER
I20260924 20:11:57.134378 18513 consensus_queue.cc:237] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } }
I20260924 20:11:57.137528 18035 catalog_manager.cc:5495] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bda497bfff04b94b7e700298ed2bb5e (127.0.115.131). New cstate: current_term: 1 leader_uuid: "6bda497bfff04b94b7e700298ed2bb5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bda497bfff04b94b7e700298ed2bb5e" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 46545 } health_report { overall_health: HEALTHY } } }
I20260924 20:11:57.157667 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.153723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.0.115.131:46545)
0924 20:11:57.153880 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:57.153883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:57.153898 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:11:57.153971 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:57.153973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:57.154025 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:57.154119 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:11:57.154467 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.155027 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.156017 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:57.156401 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:57.156523 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:57.156847 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:11:57.157272 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:11:57.157413 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:57.157449 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 20:11:57.157504 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 20:11:57.163758 18311 raft_consensus.cc:1261] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 1 FOLLOWER]: Refusing update from remote peer 6bda497bfff04b94b7e700298ed2bb5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:57.163785 18180 raft_consensus.cc:1261] T a9d56dcc3d5047de9dc310deb393133d P 10c1b158c6cb4374883a5a553d9bc4d5 [term 1 FOLLOWER]: Refusing update from remote peer 6bda497bfff04b94b7e700298ed2bb5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:57.164603 18515 consensus_queue.cc:1035] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b34719283646b69725a07759383f2d" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 44035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:57.164798 18513 consensus_queue.cc:1035] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c1b158c6cb4374883a5a553d9bc4d5" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 36989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:57.172904 18519 mvcc.cc:205] Tried to move back new op lower bound from 7332989817494614016 to 7332989817384894464. Current Snapshot: MvccSnapshot[applied={T|T < 7332989817494614016}]
I20260924 20:11:57.173053 18520 mvcc.cc:205] Tried to move back new op lower bound from 7332989817494614016 to 7332989817384894464. Current Snapshot: MvccSnapshot[applied={T|T < 7332989817494614016}]
I20260924 20:11:57.181833 18038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:57.181995 18038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:57.184439 18038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384d721ee0d342a6bdca99c754a000d1: Sending DeleteTablet for 3 replicas of tablet a9d56dcc3d5047de9dc310deb393133d
I20260924 20:11:57.185230 18291 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d56dcc3d5047de9dc310deb393133d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:57 UTC) from {username='slave'} at 127.0.0.1:50184
I20260924 20:11:57.185392 18160 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d56dcc3d5047de9dc310deb393133d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:57 UTC) from {username='slave'} at 127.0.0.1:49254
I20260924 20:11:57.185739 18528 tablet_replica.cc:330] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: stopping tablet replica
I20260924 20:11:57.186084 18528 raft_consensus.cc:2229] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:11:57.186051 18422 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d56dcc3d5047de9dc310deb393133d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:57 UTC) from {username='slave'} at 127.0.0.1:54266
I20260924 20:11:57.186360 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18098
I20260924 20:11:57.186435 18528 raft_consensus.cc:2258] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:57.186532 18530 tablet_replica.cc:330] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: stopping tablet replica
I20260924 20:11:57.186929 18530 raft_consensus.cc:2229] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:11:57.187342 18530 raft_consensus.cc:2258] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:57.188912 18528 ts_tablet_manager.cc:1905] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:57.188951 18530 ts_tablet_manager.cc:1905] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:57.192006 18530 ts_tablet_manager.cc:1918] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:57.192176 18530 log.cc:1196] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/a9d56dcc3d5047de9dc310deb393133d
I20260924 20:11:57.192350 18528 ts_tablet_manager.cc:1918] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:57.192538 18528 log.cc:1196] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/a9d56dcc3d5047de9dc310deb393133d
I20260924 20:11:57.192596 18530 ts_tablet_manager.cc:1939] T a9d56dcc3d5047de9dc310deb393133d P 6bda497bfff04b94b7e700298ed2bb5e: Deleting consensus metadata
I20260924 20:11:57.192909 18528 ts_tablet_manager.cc:1939] T a9d56dcc3d5047de9dc310deb393133d P 21b34719283646b69725a07759383f2d: Deleting consensus metadata
I20260924 20:11:57.193894 18023 catalog_manager.cc:4841] TS 6bda497bfff04b94b7e700298ed2bb5e (127.0.115.131:46545): tablet a9d56dcc3d5047de9dc310deb393133d (table test-table [id=42338b406f0040269ee31b54c4a02d44]) successfully deleted
I20260924 20:11:57.194137 18025 catalog_manager.cc:4841] TS 21b34719283646b69725a07759383f2d (127.0.115.130:44035): tablet a9d56dcc3d5047de9dc310deb393133d (table test-table [id=42338b406f0040269ee31b54c4a02d44]) successfully deleted
W20260924 20:11:57.196496 18023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:36989 (error 108)
I20260924 20:11:57.196635 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18229
I20260924 20:11:57.198186 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:57.197868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:36989 (local address 127.0.0.1:49260)
0924 20:11:57.198054 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:57.198111 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:36989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260924 20:11:57.198379 18023 catalog_manager.cc:4584] TS 10c1b158c6cb4374883a5a553d9bc4d5 (127.0.115.129:36989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9d56dcc3d5047de9dc310deb393133d: Network error: Client connection negotiation failed: client connection to 127.0.115.129:36989: connect: Connection refused (error 111)
I20260924 20:11:57.204896 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18360
I20260924 20:11:57.210642 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18007
2026-09-24T20:11:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1049 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:11:57 dist-test-slave-2pm7 krb5kdc[18534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:11:57 dist-test-slave-2pm7 krb5kdc[18534](info): set up 2 sockets
Sep 24 20:11:57 dist-test-slave-2pm7 krb5kdc[18534](info): commencing operation
krb5kdc: starting...
W20260924 20:11:58.252081 462 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:11:58 dist-test-slave-2pm7 krb5kdc[18534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:11:58.374811 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33131
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:33131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:58.485898 18547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:58.486232 18547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:58.486312 18547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:58.490342 18547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:11:58.490446 18547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:58.490494 18547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:58.490553 18547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:11:58.490594 18547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:11:58.495854 18547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:33131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:58.496996 18547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:58.498036 18547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:58.506764 18555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:58.506929 18547 server_base.cc:1034] running on GCE node
W20260924 20:11:58.506875 18552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:58.506759 18553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:58.507746 18547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:58.509075 18547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:58.510293 18547 hybrid_clock.cc:648] HybridClock initialized: now 1790280718510258 us; error 41 us; skew 500 ppm
Sep 24 20:11:58 dist-test-slave-2pm7 krb5kdc[18534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:58.517042 18547 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:11:58.517967 18547 webserver.cc:460] Webserver started at http://127.0.115.190:37029/ using document root <none> and password file <none>
I20260924 20:11:58.518544 18547 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:58.518636 18547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:58.518870 18547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:58.521091 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "04bcba76263b43bbb7503abaca56942b"
format_stamp: "Formatted at 2026-09-24 20:11:58 on dist-test-slave-2pm7"
server_key: "1e71f5d2320f9a169c5e11c7173851d7"
server_key_iv: "9ea0a42c900ac9bcf7b5087683279b9b"
server_key_version: "encryptionkey@0"
I20260924 20:11:58.521654 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "04bcba76263b43bbb7503abaca56942b"
format_stamp: "Formatted at 2026-09-24 20:11:58 on dist-test-slave-2pm7"
server_key: "1e71f5d2320f9a169c5e11c7173851d7"
server_key_iv: "9ea0a42c900ac9bcf7b5087683279b9b"
server_key_version: "encryptionkey@0"
I20260924 20:11:58.525710 18547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 20:11:58.528196 18562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:58.529554 18547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:11:58.529706 18547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "04bcba76263b43bbb7503abaca56942b"
format_stamp: "Formatted at 2026-09-24 20:11:58 on dist-test-slave-2pm7"
server_key: "1e71f5d2320f9a169c5e11c7173851d7"
server_key_iv: "9ea0a42c900ac9bcf7b5087683279b9b"
server_key_version: "encryptionkey@0"
I20260924 20:11:58.529824 18547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:58.559464 18547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:58.562577 18547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:58.562796 18547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:58.570026 18547 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:33131
I20260924 20:11:58.570042 18614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:33131 every 8 connection(s)
I20260924 20:11:58.571012 18547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:11:58.571509 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18547
I20260924 20:11:58.571652 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:11:58.571905 462 external_mini_cluster.cc:1428] Setting key 345bdff81825b03cb6743bed3d127bfd
I20260924 20:11:58.574647 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:58.581670 18615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b: Bootstrap starting.
Sep 24 20:11:58 dist-test-slave-2pm7 krb5kdc[18534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:58.583909 18615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:58.584669 18615 log.cc:824] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:58.586673 18615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b: No bootstrap required, opened a new log
I20260924 20:11:58.587100 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:58.573413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.0.115.190:33131)
0924 20:11:58.574196 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:58.574217 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:58.574275 (+ 58us) server_negotiation.cc:407] Connection header received
0924 20:11:58.575270 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:58.575296 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:58.575634 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:58.575912 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:58.576837 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:58.577830 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:58.578998 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:58.579452 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:58.579575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:58.582435 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:58.582440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:58.582453 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:58.582485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:58.584274 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:58.584705 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:58.584716 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:58.584720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:58.584872 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:58.585161 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:58.585165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:58.585167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:58.585561 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:58.585774 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:58.586049 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:11:58.586331 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":156,"threads_started":1}
I20260924 20:11:58.590849 18615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcba76263b43bbb7503abaca56942b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33131 } }
I20260924 20:11:58.591142 18615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:58.591207 18615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bcba76263b43bbb7503abaca56942b, State: Initialized, Role: FOLLOWER
I20260924 20:11:58.591701 18615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcba76263b43bbb7503abaca56942b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33131 } }
I20260924 20:11:58.591833 18615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:11:58.591881 18615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:11:58.592002 18615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:58.592834 18615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcba76263b43bbb7503abaca56942b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33131 } }
I20260924 20:11:58.593114 18615 leader_election.cc:302] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04bcba76263b43bbb7503abaca56942b; no voters:
I20260924 20:11:58.593366 18615 leader_election.cc:288] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:11:58.593511 18620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:58.593775 18620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [term 1 LEADER]: Becoming Leader. State: Replica: 04bcba76263b43bbb7503abaca56942b, State: Running, Role: LEADER
I20260924 20:11:58.594087 18620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcba76263b43bbb7503abaca56942b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33131 } }
I20260924 20:11:58.594656 18615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:11:58.595731 18621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04bcba76263b43bbb7503abaca56942b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcba76263b43bbb7503abaca56942b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33131 } } }
I20260924 20:11:58.595731 18622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04bcba76263b43bbb7503abaca56942b. Latest consensus state: current_term: 1 leader_uuid: "04bcba76263b43bbb7503abaca56942b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcba76263b43bbb7503abaca56942b" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33131 } } }
I20260924 20:11:58.595906 18621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:58.595956 18622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:58.596390 18629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:11:58.599519 18629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:11:58.604940 18629 catalog_manager.cc:1347] Generated new cluster ID: 821c4e0f476a4d04826c3992b95048d8
I20260924 20:11:58.605019 18629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:11:58.619168 18629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:11:58.620568 18629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:11:58.629338 18629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b: Generated new TSK 0
I20260924 20:11:58.630056 18629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:11:58.663179 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:58.774536 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:58.774784 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:58.774825 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:58.778951 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:58.779006 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:58.779095 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:11:58.784296 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:58.785287 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:58.786298 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:58.796017 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:58.796041 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:58.795992 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:58.796284 18641 server_base.cc:1034] running on GCE node
I20260924 20:11:58.796808 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:58.797439 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:58.798605 18641 hybrid_clock.cc:648] HybridClock initialized: now 1790280718798587 us; error 31 us; skew 500 ppm
Sep 24 20:11:58 dist-test-slave-2pm7 krb5kdc[18534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:58.801326 18641 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:11:58.802184 18641 webserver.cc:460] Webserver started at http://127.0.115.129:44547/ using document root <none> and password file <none>
I20260924 20:11:58.802757 18641 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:58.802817 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:58.803057 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:58.805228 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "0a6cd621532743b9a76ed6fce1d5765f"
format_stamp: "Formatted at 2026-09-24 20:11:58 on dist-test-slave-2pm7"
server_key: "822673061815a97f76c0f4a01b48f50f"
server_key_iv: "d4446b300b3565666dbe822a6cbb3990"
server_key_version: "encryptionkey@0"
I20260924 20:11:58.805824 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "0a6cd621532743b9a76ed6fce1d5765f"
format_stamp: "Formatted at 2026-09-24 20:11:58 on dist-test-slave-2pm7"
server_key: "822673061815a97f76c0f4a01b48f50f"
server_key_iv: "d4446b300b3565666dbe822a6cbb3990"
server_key_version: "encryptionkey@0"
I20260924 20:11:58.809767 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:11:58.812500 18656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:58.813751 18641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 20:11:58.813886 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "0a6cd621532743b9a76ed6fce1d5765f"
format_stamp: "Formatted at 2026-09-24 20:11:58 on dist-test-slave-2pm7"
server_key: "822673061815a97f76c0f4a01b48f50f"
server_key_iv: "d4446b300b3565666dbe822a6cbb3990"
server_key_version: "encryptionkey@0"
I20260924 20:11:58.814010 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:58.828012 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:58.831044 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:58.831199 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:58.831768 18641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:58.832695 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:58.832744 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:58.832787 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:58.832806 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:58.841990 18641 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:44391
I20260924 20:11:58.842059 18769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:44391 every 8 connection(s)
I20260924 20:11:58.842950 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:11:58.849534 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18641
I20260924 20:11:58.849685 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:11:58.850001 462 external_mini_cluster.cc:1428] Setting key a80c592c323f83555ceade8a3162df25
Sep 24 20:11:58 dist-test-slave-2pm7 krb5kdc[18534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:58.858173 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:58.844889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:48453 (local address 127.0.115.190:33131)
0924 20:11:58.845048 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:58.845052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:58.846018 (+ 966us) server_negotiation.cc:407] Connection header received
0924 20:11:58.847360 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:58.847363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:58.847425 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:58.847531 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:58.848758 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:58.849338 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:58.850764 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:58.851013 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:58.851126 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:58.854139 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:58.854143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:58.854145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:58.854173 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:58.855863 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:58.856476 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:58.856480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:58.856482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:58.856568 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:58.856957 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:58.856962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:58.856963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:58.857163 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:58.857281 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:58.857841 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 20:11:58.857979 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:11:58.859103 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:58.845290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33131 (local address 127.0.115.129:48453)
0924 20:11:58.845758 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:58.845801 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:58.847138 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:58.847715 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:58.847722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:58.848097 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:58.848596 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:58.848610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:58.849486 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:58.849490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:58.850638 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:58.850648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:58.851169 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:58.851173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:58.851343 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:58.852009 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:58.852036 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:58.853932 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:58.856045 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:58.856052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:58.856069 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:58.856358 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:58.856677 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:58.856682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:58.856684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:58.856850 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:58.857280 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:58.857287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:58.857567 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 20:11:58.857774 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 20:11:58.858146 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:11:58.860594 18770 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33131
I20260924 20:11:58.860865 18770 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:58.861451 18770 heartbeater.cc:511] Master 127.0.115.190:33131 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:11:58.863327 18579 ts_manager.cc:194] Registered new tserver with Master: 0a6cd621532743b9a76ed6fce1d5765f (127.0.115.129:44391)
I20260924 20:11:58.864661 18579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:48453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:11:58.878903 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:58.993682 18775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:58.993942 18775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:58.993989 18775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:58.998440 18775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:58.998505 18775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:58.998564 18775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:11:59.003933 18775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:59.005101 18775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:59.006130 18775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:59.016080 18780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:59.016113 18783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:59.016127 18781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:59.016362 18775 server_base.cc:1034] running on GCE node
I20260924 20:11:59.016834 18775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:59.017514 18775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:59.018697 18775 hybrid_clock.cc:648] HybridClock initialized: now 1790280719018662 us; error 44 us; skew 500 ppm
Sep 24 20:11:59 dist-test-slave-2pm7 krb5kdc[18534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:59.021394 18775 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:11:59.022414 18775 webserver.cc:460] Webserver started at http://127.0.115.130:34403/ using document root <none> and password file <none>
I20260924 20:11:59.022998 18775 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:59.023093 18775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:59.023329 18775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:59.025502 18775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "7a44d61c47b04df19f943355ba2902d1"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "66ccc1a611f7adb35c0745a1339d1406"
server_key_iv: "77bcf682aed5d57d034ebf49e177e6f2"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.026191 18775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "7a44d61c47b04df19f943355ba2902d1"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "66ccc1a611f7adb35c0745a1339d1406"
server_key_iv: "77bcf682aed5d57d034ebf49e177e6f2"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.029901 18775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:11:59.032393 18790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.033725 18775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:11:59.033854 18775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "7a44d61c47b04df19f943355ba2902d1"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "66ccc1a611f7adb35c0745a1339d1406"
server_key_iv: "77bcf682aed5d57d034ebf49e177e6f2"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.033973 18775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:59.048420 18775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:59.051453 18775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:59.051673 18775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:59.052371 18775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:59.053264 18775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:59.053337 18775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.053411 18775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:59.053463 18775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.062275 18775 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:36147
I20260924 20:11:59.062299 18903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:36147 every 8 connection(s)
I20260924 20:11:59.063177 18775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:11:59.065546 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18775
I20260924 20:11:59.065692 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:11:59.066052 462 external_mini_cluster.cc:1428] Setting key 4ce6eb8c3bdd8799762d6f8b19b73e2c
Sep 24 20:11:59 dist-test-slave-2pm7 krb5kdc[18534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
I20260924 20:11:59.078094 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.065114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:36999 (local address 127.0.115.190:33131)
0924 20:11:59.065279 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.065284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.066310 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 20:11:59.067827 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.067830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.067886 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.068060 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:59.069241 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.069774 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.070911 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.071145 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.071271 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.074178 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:59.074181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:59.074183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:59.074206 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:59.075746 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:59.076370 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:59.076372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:59.076374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:59.076448 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:59.076810 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:59.076814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:59.076815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:59.077013 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:59.077120 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.077749 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.077904 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:11:59.079144 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.065608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33131 (local address 127.0.115.130:36999)
0924 20:11:59.066141 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.066200 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.067610 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.068242 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.068249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.068614 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:59.069097 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.069110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.069897 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.069901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.070785 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.070793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.071346 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.071353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.071596 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.072290 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:59.072320 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:59.074004 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:59.075861 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:59.075868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:59.075885 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:59.076217 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:59.076545 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:59.076549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:59.076551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:59.076702 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:59.077130 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:59.077139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:59.077504 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.077706 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.078048 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260924 20:11:59.080379 18904 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33131
I20260924 20:11:59.080626 18904 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:59.081147 18904 heartbeater.cc:511] Master 127.0.115.190:33131 requested a full tablet report, sending...
I20260924 20:11:59.082356 18579 ts_manager.cc:194] Registered new tserver with Master: 7a44d61c47b04df19f943355ba2902d1 (127.0.115.130:36147)
I20260924 20:11:59.083263 18579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:36999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:11:59.096225 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33131
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:11:59.207229 18909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:59.207485 18909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:59.207526 18909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:59.211509 18909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:59.211567 18909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:59.211624 18909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:11:59.217022 18909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:33131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:59.218319 18909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:59.219398 18909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:59.229259 18915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:59.229333 18914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:59.229318 18917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:59.229576 18909 server_base.cc:1034] running on GCE node
I20260924 20:11:59.229993 18909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:59.230618 18909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:59.231802 18909 hybrid_clock.cc:648] HybridClock initialized: now 1790280719231771 us; error 52 us; skew 500 ppm
Sep 24 20:11:59 dist-test-slave-2pm7 krb5kdc[18534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:11:59.234648 18909 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:11:59.235531 18909 webserver.cc:460] Webserver started at http://127.0.115.131:42111/ using document root <none> and password file <none>
I20260924 20:11:59.236152 18909 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:59.236240 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:59.236464 18909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:59.238590 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "8ea097cce6874ea7a5619d2a342597b6"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "d0248458d52b15434d8a125c9c6328ae"
server_key_iv: "e66003f6c66c63e88dc617a870546671"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.239149 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "8ea097cce6874ea7a5619d2a342597b6"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "d0248458d52b15434d8a125c9c6328ae"
server_key_iv: "e66003f6c66c63e88dc617a870546671"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.242808 18909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 20:11:59.245194 18924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.246337 18909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:11:59.246471 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "8ea097cce6874ea7a5619d2a342597b6"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "d0248458d52b15434d8a125c9c6328ae"
server_key_iv: "e66003f6c66c63e88dc617a870546671"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.246589 18909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:59.266363 18909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:59.269788 18909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:59.270013 18909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:59.270658 18909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:59.271541 18909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:59.271615 18909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.271692 18909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:59.271736 18909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.280658 18909 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:40505
I20260924 20:11:59.280679 19037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:40505 every 8 connection(s)
I20260924 20:11:59.281569 18909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:11:59.282739 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 18909
I20260924 20:11:59.282884 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:11:59.283255 462 external_mini_cluster.cc:1428] Setting key fa0eae72ff013f6967a03876b6490284
Sep 24 20:11:59 dist-test-slave-2pm7 krb5kdc[18534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:11:59.294924 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.283603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:51691 (local address 127.0.115.190:33131)
0924 20:11:59.283773 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.283778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.284790 (+ 1012us) server_negotiation.cc:407] Connection header received
0924 20:11:59.286234 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.286236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.286280 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.286345 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:59.287420 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.287930 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.289050 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.289257 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.289331 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.291565 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:59.291568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:59.291570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:59.291594 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:59.293026 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:59.293453 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:59.293455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:59.293456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:59.293535 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:59.293893 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:59.293896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:59.293897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:59.294063 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:59.294181 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.294709 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.294801 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:11:59.295943 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.284105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33131 (local address 127.0.115.131:51691)
0924 20:11:59.284632 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.284687 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.286051 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.286476 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.286482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.286822 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:59.287303 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.287316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.288083 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.288086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.288949 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.288958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.289381 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.289384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.289521 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.290032 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:11:59.290048 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:59.291396 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:59.293123 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:59.293127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:59.293137 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:59.293370 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:59.293640 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:11:59.293643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:11:59.293644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:11:59.293804 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:11:59.294206 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:59.294218 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:59.294498 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.294661 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.294930 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":137,"threads_started":1}
I20260924 20:11:59.297262 19038 heartbeater.cc:347] Connected to a master server at 127.0.115.190:33131
I20260924 20:11:59.297542 19038 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:59.298041 19038 heartbeater.cc:511] Master 127.0.115.190:33131 requested a full tablet report, sending...
I20260924 20:11:59.299047 18579 ts_manager.cc:194] Registered new tserver with Master: 8ea097cce6874ea7a5619d2a342597b6 (127.0.115.131:40505)
I20260924 20:11:59.299624 18579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:51691
I20260924 20:11:59.309417 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:11:59.319164 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.311450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.0.115.190:33131)
0924 20:11:59.311575 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.311579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.311755 (+ 176us) server_negotiation.cc:407] Connection header received
0924 20:11:59.311907 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.311909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.312014 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.312103 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:59.312827 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.313330 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.314255 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.314468 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.314548 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.316058 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:59.316061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:59.316064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:11:59.316087 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:59.317501 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:59.318021 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:59.318024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:59.318025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:59.318096 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:11:59.318368 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:11:59.318370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:11:59.318371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:11:59.318577 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:59.318687 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.318962 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.319042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:11:59.322122 18579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:11:59.325384 18579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:11:59.339720 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.333701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:36147 (local address 127.0.0.1:41130)
0924 20:11:59.334042 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.334095 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.334261 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.335157 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.335165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.335190 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:59.335304 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.335311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.336744 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.336746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.338129 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.338140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.339201 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.339204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.339312 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.339357 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.339444 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.339557 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260924 20:11:59.339892 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.334503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:40505 (local address 127.0.0.1:53338)
0924 20:11:59.334730 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.334745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.334830 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.336025 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.336028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.336047 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:59.336149 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.336157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.337493 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.337497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.338598 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.338608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.339520 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.339524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.339607 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.339629 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.339693 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.339759 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260924 20:11:59.339982 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.335171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.0.115.131:40505)
0924 20:11:59.335536 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.335548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.335572 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:11:59.335649 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.335654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.335843 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.336047 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:59.336246 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.337330 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.338719 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.339391 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.339517 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.339670 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.339725 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.339817 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":171,"threads_started":1}
I20260924 20:11:59.340431 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.334840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:44391 (local address 127.0.0.1:49546)
0924 20:11:59.335043 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.335058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.335142 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.336442 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.336445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.336463 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:59.336542 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.336548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.337778 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.337785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.338928 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.338939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.340044 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.340048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.340137 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.340161 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.340207 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.340290 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":68,"thread_start_us":88,"threads_started":1}
I20260924 20:11:59.341063 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.335685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.0.115.129:44391)
0924 20:11:59.335998 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.336012 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.336034 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:11:59.336107 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.336112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.336302 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.336438 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:59.336665 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.337629 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.339065 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.339865 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.340616 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.340771 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.340827 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.340918 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260924 20:11:59.341063 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.333824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.115.130:36147)
0924 20:11:59.334185 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.334198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.334224 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:11:59.334513 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.334519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.334758 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.334944 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:59.335434 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.336547 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.338266 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.339064 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.340670 (+ 1606us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.340788 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.340833 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.340918 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":133,"threads_started":1}
I20260924 20:11:59.343416 18838 tablet_service.cc:1467] Processing CreateTablet for tablet c12a6a97794242d18a9910ad5406f1ca (DEFAULT_TABLE table=test-table [id=a11c56d083174aea93e3e4ede18111f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:59.343441 18972 tablet_service.cc:1467] Processing CreateTablet for tablet c12a6a97794242d18a9910ad5406f1ca (DEFAULT_TABLE table=test-table [id=a11c56d083174aea93e3e4ede18111f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:59.343643 18704 tablet_service.cc:1467] Processing CreateTablet for tablet c12a6a97794242d18a9910ad5406f1ca (DEFAULT_TABLE table=test-table [id=a11c56d083174aea93e3e4ede18111f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:11:59.344551 18838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12a6a97794242d18a9910ad5406f1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:59.344551 18704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12a6a97794242d18a9910ad5406f1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:59.344554 18972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12a6a97794242d18a9910ad5406f1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:59.351272 19055 tablet_bootstrap.cc:492] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Bootstrap starting.
I20260924 20:11:59.352548 19056 tablet_bootstrap.cc:492] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Bootstrap starting.
I20260924 20:11:59.353559 19055 tablet_bootstrap.cc:654] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:59.354614 19056 tablet_bootstrap.cc:654] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:59.354650 19055 log.cc:824] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:59.355288 19057 tablet_bootstrap.cc:492] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f: Bootstrap starting.
I20260924 20:11:59.355468 19056 log.cc:824] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:59.356838 19055 tablet_bootstrap.cc:492] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: No bootstrap required, opened a new log
I20260924 20:11:59.357040 19055 ts_tablet_manager.cc:1397] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:11:59.357398 19056 tablet_bootstrap.cc:492] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: No bootstrap required, opened a new log
I20260924 20:11:59.357416 19057 tablet_bootstrap.cc:654] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:59.357570 19056 ts_tablet_manager.cc:1397] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:11:59.358420 19057 log.cc:824] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:59.360132 19057 tablet_bootstrap.cc:492] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f: No bootstrap required, opened a new log
I20260924 20:11:59.360320 19057 ts_tablet_manager.cc:1397] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:11:59.361057 19055 raft_consensus.cc:348] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.361361 19055 raft_consensus.cc:374] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:59.361430 19055 raft_consensus.cc:729] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a44d61c47b04df19f943355ba2902d1, State: Initialized, Role: FOLLOWER
I20260924 20:11:59.361276 19056 raft_consensus.cc:348] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.361526 19056 raft_consensus.cc:374] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:59.361578 19056 raft_consensus.cc:729] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea097cce6874ea7a5619d2a342597b6, State: Initialized, Role: FOLLOWER
I20260924 20:11:59.361945 19056 consensus_queue.cc:260] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.361945 19055 consensus_queue.cc:260] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.362959 19056 ts_tablet_manager.cc:1428] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:11:59.362959 19055 ts_tablet_manager.cc:1428] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:11:59.363339 19038 heartbeater.cc:503] Master 127.0.115.190:33131 was elected leader, sending a full tablet report...
I20260924 20:11:59.363641 18904 heartbeater.cc:503] Master 127.0.115.190:33131 was elected leader, sending a full tablet report...
I20260924 20:11:59.363727 19057 raft_consensus.cc:348] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.363979 19057 raft_consensus.cc:374] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:59.364030 19057 raft_consensus.cc:729] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6cd621532743b9a76ed6fce1d5765f, State: Initialized, Role: FOLLOWER
I20260924 20:11:59.364459 19057 consensus_queue.cc:260] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.365410 19057 ts_tablet_manager.cc:1428] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:11:59.365774 18770 heartbeater.cc:503] Master 127.0.115.190:33131 was elected leader, sending a full tablet report...
I20260924 20:11:59.376245 19061 raft_consensus.cc:482] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:11:59.376411 19061 raft_consensus.cc:504] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.377480 19061 leader_election.cc:288] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a44d61c47b04df19f943355ba2902d1 (127.0.115.130:36147), 0a6cd621532743b9a76ed6fce1d5765f (127.0.115.129:44391)
I20260924 20:11:59.381529 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.377842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:51033 (local address 127.0.115.130:36147)
0924 20:11:59.377984 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.377991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.378008 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:11:59.378058 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.378061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.378108 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.378182 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:59.378586 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.379239 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.380470 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.380926 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.381027 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.381068 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.381307 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.381399 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:11:59.381537 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.377695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:36147 (local address 127.0.115.131:51033)
0924 20:11:59.377821 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.377833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.377941 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.378304 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.378309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.378328 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:59.378456 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.378465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.379349 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.379352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.380334 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.380344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.381049 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.381053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.381168 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.381193 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.381268 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.381341 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 20:11:59.381628 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.378155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:51579 (local address 127.0.115.129:44391)
0924 20:11:59.378288 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.378294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.378404 (+ 110us) server_negotiation.cc:407] Connection header received
0924 20:11:59.378493 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.378499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.378560 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.378634 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:11:59.378925 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.379494 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.380607 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.381052 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.381168 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.381218 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.381430 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.381504 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:11:59.381790 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.377998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:44391 (local address 127.0.115.131:51579)
0924 20:11:59.378323 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.378334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.378419 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.378665 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.378669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.378687 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:11:59.378808 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.378815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.379606 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.379609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.380475 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.380494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.381175 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.381178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.381293 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.381318 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.381393 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.381456 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260924 20:11:59.382400 18858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12a6a97794242d18a9910ad5406f1ca" candidate_uuid: "8ea097cce6874ea7a5619d2a342597b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a44d61c47b04df19f943355ba2902d1" is_pre_election: true
I20260924 20:11:59.382465 18724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12a6a97794242d18a9910ad5406f1ca" candidate_uuid: "8ea097cce6874ea7a5619d2a342597b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6cd621532743b9a76ed6fce1d5765f" is_pre_election: true
I20260924 20:11:59.382758 18858 raft_consensus.cc:2454] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea097cce6874ea7a5619d2a342597b6 in term 0.
I20260924 20:11:59.382781 18724 raft_consensus.cc:2454] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea097cce6874ea7a5619d2a342597b6 in term 0.
I20260924 20:11:59.383235 18926 leader_election.cc:302] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a6cd621532743b9a76ed6fce1d5765f, 8ea097cce6874ea7a5619d2a342597b6; no voters:
I20260924 20:11:59.383522 19061 raft_consensus.cc:2781] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:11:59.383620 19061 raft_consensus.cc:482] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:11:59.383679 19061 raft_consensus.cc:3037] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:59.384825 19061 raft_consensus.cc:504] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.385200 19061 leader_election.cc:288] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [CANDIDATE]: Term 1 election: Requested vote from peers 7a44d61c47b04df19f943355ba2902d1 (127.0.115.130:36147), 0a6cd621532743b9a76ed6fce1d5765f (127.0.115.129:44391)
I20260924 20:11:59.385748 18858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12a6a97794242d18a9910ad5406f1ca" candidate_uuid: "8ea097cce6874ea7a5619d2a342597b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a44d61c47b04df19f943355ba2902d1"
I20260924 20:11:59.385788 18724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12a6a97794242d18a9910ad5406f1ca" candidate_uuid: "8ea097cce6874ea7a5619d2a342597b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6cd621532743b9a76ed6fce1d5765f"
I20260924 20:11:59.385949 18858 raft_consensus.cc:3037] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:59.385934 18724 raft_consensus.cc:3037] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:59.387073 18858 raft_consensus.cc:2454] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea097cce6874ea7a5619d2a342597b6 in term 1.
I20260924 20:11:59.387073 18724 raft_consensus.cc:2454] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea097cce6874ea7a5619d2a342597b6 in term 1.
I20260924 20:11:59.387445 18926 leader_election.cc:302] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a44d61c47b04df19f943355ba2902d1, 8ea097cce6874ea7a5619d2a342597b6; no voters:
I20260924 20:11:59.387653 19061 raft_consensus.cc:2781] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:59.387923 19061 raft_consensus.cc:686] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 1 LEADER]: Becoming Leader. State: Replica: 8ea097cce6874ea7a5619d2a342597b6, State: Running, Role: LEADER
I20260924 20:11:59.388340 19061 consensus_queue.cc:237] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } }
I20260924 20:11:59.391603 18579 catalog_manager.cc:5495] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ea097cce6874ea7a5619d2a342597b6 (127.0.115.131). New cstate: current_term: 1 leader_uuid: "8ea097cce6874ea7a5619d2a342597b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea097cce6874ea7a5619d2a342597b6" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 40505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 } health_report { overall_health: UNKNOWN } } }
I20260924 20:11:59.403707 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.400010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.0.115.131:40505)
0924 20:11:59.400148 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.400152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.400166 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:11:59.400326 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.400334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.400394 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.400516 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:11:59.400769 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.401290 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.402182 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.402409 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.402524 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.402821 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:11:59.403243 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:11:59.403410 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.403513 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.403551 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:11:59.409781 18724 raft_consensus.cc:1261] T c12a6a97794242d18a9910ad5406f1ca P 0a6cd621532743b9a76ed6fce1d5765f [term 1 FOLLOWER]: Refusing update from remote peer 8ea097cce6874ea7a5619d2a342597b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:59.409797 18858 raft_consensus.cc:1261] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 1 FOLLOWER]: Refusing update from remote peer 8ea097cce6874ea7a5619d2a342597b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:11:59.410620 19065 consensus_queue.cc:1035] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a6cd621532743b9a76ed6fce1d5765f" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 44391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:59.410867 19061 consensus_queue.cc:1035] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a44d61c47b04df19f943355ba2902d1" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 36147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:11:59.418624 19069 mvcc.cc:205] Tried to move back new op lower bound from 7332989826694504448 to 7332989826617344000. Current Snapshot: MvccSnapshot[applied={T|T < 7332989826694504448}]
I20260924 20:11:59.420266 19070 mvcc.cc:205] Tried to move back new op lower bound from 7332989826694504448 to 7332989826617344000. Current Snapshot: MvccSnapshot[applied={T|T < 7332989826694504448}]
I20260924 20:11:59.427348 18577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:59.427539 18577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:11:59.430078 18577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04bcba76263b43bbb7503abaca56942b: Sending DeleteTablet for 3 replicas of tablet c12a6a97794242d18a9910ad5406f1ca
I20260924 20:11:59.431200 18704 tablet_service.cc:1514] Processing DeleteTablet for tablet c12a6a97794242d18a9910ad5406f1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:59 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:49546
I20260924 20:11:59.431201 18838 tablet_service.cc:1514] Processing DeleteTablet for tablet c12a6a97794242d18a9910ad5406f1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:59 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:41130
I20260924 20:11:59.431563 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18641
I20260924 20:11:59.431766 18972 tablet_service.cc:1514] Processing DeleteTablet for tablet c12a6a97794242d18a9910ad5406f1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:11:59 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:53338
I20260924 20:11:59.432494 19079 tablet_replica.cc:330] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: stopping tablet replica
I20260924 20:11:59.432946 19079 raft_consensus.cc:2229] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:11:59.433103 19080 tablet_replica.cc:330] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: stopping tablet replica
I20260924 20:11:59.433379 19080 raft_consensus.cc:2229] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:11:59.433406 19079 raft_consensus.cc:2258] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:59.433902 19080 raft_consensus.cc:2258] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:11:59.435274 19079 ts_tablet_manager.cc:1905] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:59.436758 19080 ts_tablet_manager.cc:1905] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:11:59.438551 19079 ts_tablet_manager.cc:1918] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:59.438692 19079 log.cc:1196] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/c12a6a97794242d18a9910ad5406f1ca
I20260924 20:11:59.439057 19079 ts_tablet_manager.cc:1939] T c12a6a97794242d18a9910ad5406f1ca P 7a44d61c47b04df19f943355ba2902d1: Deleting consensus metadata
I20260924 20:11:59.440171 19080 ts_tablet_manager.cc:1918] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:11:59.440328 19080 log.cc:1196] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/c12a6a97794242d18a9910ad5406f1ca
I20260924 20:11:59.440433 18564 catalog_manager.cc:4841] TS 7a44d61c47b04df19f943355ba2902d1 (127.0.115.130:36147): tablet c12a6a97794242d18a9910ad5406f1ca (table test-table [id=a11c56d083174aea93e3e4ede18111f2]) successfully deleted
I20260924 20:11:59.440755 19080 ts_tablet_manager.cc:1939] T c12a6a97794242d18a9910ad5406f1ca P 8ea097cce6874ea7a5619d2a342597b6: Deleting consensus metadata
I20260924 20:11:59.441964 18564 catalog_manager.cc:4841] TS 8ea097cce6874ea7a5619d2a342597b6 (127.0.115.131:40505): tablet c12a6a97794242d18a9910ad5406f1ca (table test-table [id=a11c56d083174aea93e3e4ede18111f2]) successfully deleted
W20260924 20:11:59.443794 18564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:44391 (error 108)
I20260924 20:11:59.444729 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18775
I20260924 20:11:59.444977 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.444618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:44391 (local address 127.0.0.1:49554)
0924 20:11:59.444787 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.444871 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:44391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 20:11:59.446528 18564 catalog_manager.cc:4584] TS 0a6cd621532743b9a76ed6fce1d5765f (127.0.115.129:44391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c12a6a97794242d18a9910ad5406f1ca: Network error: Client connection negotiation failed: client connection to 127.0.115.129:44391: connect: Connection refused (error 111)
I20260924 20:11:59.451161 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18909
I20260924 20:11:59.456727 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 18547
2026-09-24T20:11:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2246 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T20:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:11:59Z Disabled control of system clock
I20260924 20:11:59.490263 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:40055
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:40055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:11:59.603385 19085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:59.603658 19085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:59.603693 19085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:59.607998 19085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:11:59.608062 19085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:59.608078 19085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:59.608094 19085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:11:59.608110 19085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:11:59.613037 19085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:40055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:40055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:59.614171 19085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:59.615675 19085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:59.627457 19093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:59.627485 19090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:59.627463 19091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:59.627874 19085 server_base.cc:1034] running on GCE node
I20260924 20:11:59.628773 19085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:59.630293 19085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:59.631510 19085 hybrid_clock.cc:648] HybridClock initialized: now 1790280719631474 us; error 49 us; skew 500 ppm
I20260924 20:11:59.633906 19085 webserver.cc:460] Webserver started at http://127.0.115.190:36395/ using document root <none> and password file <none>
I20260924 20:11:59.634682 19085 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:59.634774 19085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:59.635022 19085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:59.637918 19085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "d5a0ef8092334826b7eaa392dab01b4a"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "3e5ae380443dc123c81347d837217945"
server_key_iv: "40b726225d4efee25710c1641fe274d8"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.638657 19085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "d5a0ef8092334826b7eaa392dab01b4a"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "3e5ae380443dc123c81347d837217945"
server_key_iv: "40b726225d4efee25710c1641fe274d8"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.643636 19085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 20:11:59.646878 19099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.648465 19085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 20:11:59.648613 19085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "d5a0ef8092334826b7eaa392dab01b4a"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "3e5ae380443dc123c81347d837217945"
server_key_iv: "40b726225d4efee25710c1641fe274d8"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.648744 19085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:59.665592 19085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:59.666432 19085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:59.666647 19085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:59.675652 19151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:40055 every 8 connection(s)
I20260924 20:11:59.675652 19085 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:40055
I20260924 20:11:59.676923 19085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:11:59.680241 19152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:11:59.680488 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19085
I20260924 20:11:59.680615 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:11:59.680924 462 external_mini_cluster.cc:1428] Setting key 1470c9aa6e17eb09e2396df21d0b536f
I20260924 20:11:59.686566 19152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a: Bootstrap starting.
I20260924 20:11:59.689076 19152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a: Neither blocks nor log segments found. Creating new log.
I20260924 20:11:59.689917 19152 log.cc:824] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:11:59.692375 19152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a: No bootstrap required, opened a new log
I20260924 20:11:59.692611 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.682412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.0.115.190:40055)
0924 20:11:59.683172 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.683193 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.683241 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:11:59.684323 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.684350 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.684906 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.685277 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:59.685769 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.687386 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.688646 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.689444 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.689625 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.690514 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:59.690520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:59.690541 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:11:59.690997 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:59.691064 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:11:59.691110 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:59.691210 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.691395 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.691661 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":119,"threads_started":1}
I20260924 20:11:59.696286 19152 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a0ef8092334826b7eaa392dab01b4a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40055 } }
I20260924 20:11:59.696502 19152 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:11:59.696564 19152 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a0ef8092334826b7eaa392dab01b4a, State: Initialized, Role: FOLLOWER
I20260924 20:11:59.697002 19152 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a0ef8092334826b7eaa392dab01b4a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40055 } }
I20260924 20:11:59.697149 19152 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:11:59.697214 19152 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:11:59.697315 19152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:11:59.698210 19152 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a0ef8092334826b7eaa392dab01b4a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40055 } }
I20260924 20:11:59.698560 19152 leader_election.cc:302] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a0ef8092334826b7eaa392dab01b4a; no voters:
I20260924 20:11:59.698868 19152 leader_election.cc:288] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:11:59.698987 19157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:11:59.699240 19157 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [term 1 LEADER]: Becoming Leader. State: Replica: d5a0ef8092334826b7eaa392dab01b4a, State: Running, Role: LEADER
I20260924 20:11:59.700089 19157 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a0ef8092334826b7eaa392dab01b4a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40055 } }
I20260924 20:11:59.700273 19152 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:11:59.701762 19159 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a0ef8092334826b7eaa392dab01b4a. Latest consensus state: current_term: 1 leader_uuid: "d5a0ef8092334826b7eaa392dab01b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a0ef8092334826b7eaa392dab01b4a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40055 } } }
I20260924 20:11:59.701897 19159 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:59.702267 19158 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a0ef8092334826b7eaa392dab01b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a0ef8092334826b7eaa392dab01b4a" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40055 } } }
I20260924 20:11:59.702309 19168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:11:59.702363 19158 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a [sys.catalog]: This master's current role is: LEADER
I20260924 20:11:59.705699 19168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:11:59.712320 19168 catalog_manager.cc:1347] Generated new cluster ID: 03760715b9664d0faa72e394529cc131
I20260924 20:11:59.712389 19168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:11:59.731168 19168 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:11:59.732204 19168 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:11:59.737857 19168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a: Generated new TSK 0
I20260924 20:11:59.738736 19168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:11:59.750489 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40055
--builtin_ntp_servers=127.0.115.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:11:59.864969 19176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:11:59.865229 19176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:11:59.865267 19176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:11:59.869314 19176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:11:59.869371 19176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:11:59.869424 19176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:11:59.874071 19176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:11:59.875121 19176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:11:59.876665 19176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:11:59.886485 19181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:59.886531 19176 server_base.cc:1034] running on GCE node
W20260924 20:11:59.886498 19182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:11:59.886740 19184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:11:59.887140 19176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:11:59.887779 19176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:11:59.888939 19176 hybrid_clock.cc:648] HybridClock initialized: now 1790280719888916 us; error 38 us; skew 500 ppm
I20260924 20:11:59.890684 19176 webserver.cc:460] Webserver started at http://127.0.115.129:33449/ using document root <none> and password file <none>
I20260924 20:11:59.891243 19176 fs_manager.cc:359] Metadata directory not provided
I20260924 20:11:59.891302 19176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:11:59.891574 19176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:11:59.893651 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "498bb557de6243558a000a9433d371cf"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "5cd6899a3be0e74138da252eedf81a8a"
server_key_iv: "421c3c5c5eba8d38d6302955aeea1577"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.894199 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "498bb557de6243558a000a9433d371cf"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "5cd6899a3be0e74138da252eedf81a8a"
server_key_iv: "421c3c5c5eba8d38d6302955aeea1577"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.898221 19176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 20:11:59.901113 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.902603 19176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:11:59.902750 19176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "498bb557de6243558a000a9433d371cf"
format_stamp: "Formatted at 2026-09-24 20:11:59 on dist-test-slave-2pm7"
server_key: "5cd6899a3be0e74138da252eedf81a8a"
server_key_iv: "421c3c5c5eba8d38d6302955aeea1577"
server_key_version: "encryptionkey@0"
I20260924 20:11:59.902879 19176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:11:59.926381 19176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:11:59.927152 19176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:11:59.927377 19176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:11:59.928089 19176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:11:59.929100 19176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:11:59.929177 19176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.929250 19176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:11:59.929302 19176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:11:59.938803 19176 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:41293
I20260924 20:11:59.938822 19303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:41293 every 8 connection(s)
I20260924 20:11:59.939743 19176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:11:59.947424 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19176
I20260924 20:11:59.947582 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:11:59.947875 462 external_mini_cluster.cc:1428] Setting key 76fca3b011cacd6b12f00f04c7d230a0
I20260924 20:11:59.949255 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.941651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:58933 (local address 127.0.115.190:40055)
0924 20:11:59.941813 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:11:59.941818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:11:59.942985 (+ 1167us) server_negotiation.cc:407] Connection header received
0924 20:11:59.944479 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:11:59.944482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:11:59.944530 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:11:59.944607 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:11:59.945485 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.946019 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.947106 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.947319 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.947464 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.948741 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:11:59.948745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:11:59.948748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:11:59.948766 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:11:59.948785 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:11:59.948793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:11:59.948874 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 20:11:59.949049 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 20:11:59.949114 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:11:59.949966 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40055
--builtin_ntp_servers=127.0.115.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:11:59.950160 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:11:59.942129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:40055 (local address 127.0.115.129:58933)
0924 20:11:59.942839 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 20:11:59.942880 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:11:59.944211 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:11:59.944778 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:11:59.944784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:11:59.944868 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:11:59.945354 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.945366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.946126 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.946128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.947002 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:11:59.947010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:11:59.947504 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:11:59.947508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:11:59.947717 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:11:59.948431 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:11:59.948434 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:11:59.948491 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:11:59.948589 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:11:59.948877 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:11:59.948889 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:11:59.948895 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:11:59.949038 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 20:11:59.949316 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":115,"threads_started":1}
I20260924 20:11:59.951360 19304 heartbeater.cc:347] Connected to a master server at 127.0.115.190:40055
I20260924 20:11:59.951613 19304 heartbeater.cc:464] Registering TS with master...
I20260924 20:11:59.952203 19304 heartbeater.cc:511] Master 127.0.115.190:40055 requested a full tablet report, sending...
I20260924 20:11:59.953696 19116 ts_manager.cc:194] Registered new tserver with Master: 498bb557de6243558a000a9433d371cf (127.0.115.129:41293)
I20260924 20:11:59.954936 19116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.129:58933
W20260924 20:12:00.061970 19308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:00.062316 19308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:00.062388 19308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:00.066670 19308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:00.066767 19308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:00.066865 19308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:12:00.071895 19308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:00.073201 19308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:00.074700 19308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:00.084630 19314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:00.084764 19313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:00.084659 19316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:00.084674 19308 server_base.cc:1034] running on GCE node
I20260924 20:12:00.085237 19308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:00.085853 19308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:00.087018 19308 hybrid_clock.cc:648] HybridClock initialized: now 1790280720087003 us; error 25 us; skew 500 ppm
I20260924 20:12:00.088871 19308 webserver.cc:460] Webserver started at http://127.0.115.130:37499/ using document root <none> and password file <none>
I20260924 20:12:00.089465 19308 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:00.089526 19308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:00.089766 19308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:00.091836 19308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "1766c1031f264eb880e2778bf7b59af7"
format_stamp: "Formatted at 2026-09-24 20:12:00 on dist-test-slave-2pm7"
server_key: "676ca2a1c7447bb9f262f8656f5631f9"
server_key_iv: "ce5097ed9b197d01a0b51070aea075c6"
server_key_version: "encryptionkey@0"
I20260924 20:12:00.092465 19308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "1766c1031f264eb880e2778bf7b59af7"
format_stamp: "Formatted at 2026-09-24 20:12:00 on dist-test-slave-2pm7"
server_key: "676ca2a1c7447bb9f262f8656f5631f9"
server_key_iv: "ce5097ed9b197d01a0b51070aea075c6"
server_key_version: "encryptionkey@0"
I20260924 20:12:00.096113 19308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:12:00.098465 19322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:00.099627 19308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:00.099751 19308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "1766c1031f264eb880e2778bf7b59af7"
format_stamp: "Formatted at 2026-09-24 20:12:00 on dist-test-slave-2pm7"
server_key: "676ca2a1c7447bb9f262f8656f5631f9"
server_key_iv: "ce5097ed9b197d01a0b51070aea075c6"
server_key_version: "encryptionkey@0"
I20260924 20:12:00.099855 19308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:00.110476 19308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:00.111112 19308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:00.111303 19308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:00.111877 19308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:00.112872 19308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:00.112918 19308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:00.112989 19308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:00.113023 19308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:00.121541 19308 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:37383
I20260924 20:12:00.121564 19435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:37383 every 8 connection(s)
I20260924 20:12:00.122444 19308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:12:00.126700 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19308
I20260924 20:12:00.126850 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:12:00.127166 462 external_mini_cluster.cc:1428] Setting key 4d46888bed6e5193d848d24f457c1bd3
I20260924 20:12:00.129297 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40055
--builtin_ntp_servers=127.0.115.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:12:00.133108 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.124343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35319 (local address 127.0.115.190:40055)
0924 20:12:00.124498 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.124502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.125591 (+ 1089us) server_negotiation.cc:407] Connection header received
0924 20:12:00.127187 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.127190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.127267 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.127353 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:00.128562 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.129149 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.130926 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.131306 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.131426 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.132529 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:00.132535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:00.132541 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:00.132558 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:00.132585 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:00.132598 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:00.132677 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.132819 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.132890 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:12:00.133870 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.124806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:40055 (local address 127.0.115.130:35319)
0924 20:12:00.125447 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.125487 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:00.126950 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:00.127563 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:00.127573 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:00.127672 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:00.128340 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.128385 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.129288 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.129292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.130793 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.130808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.131454 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.131457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.131600 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.132273 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:12:00.132275 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:00.132315 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:12:00.132378 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:00.132685 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:00.132691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:00.132696 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:12:00.132788 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 20:12:00.133011 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260924 20:12:00.135089 19436 heartbeater.cc:347] Connected to a master server at 127.0.115.190:40055
I20260924 20:12:00.135317 19436 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:00.135834 19436 heartbeater.cc:511] Master 127.0.115.190:40055 requested a full tablet report, sending...
I20260924 20:12:00.136986 19116 ts_manager.cc:194] Registered new tserver with Master: 1766c1031f264eb880e2778bf7b59af7 (127.0.115.130:37383)
I20260924 20:12:00.137573 19116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.130:35319
W20260924 20:12:00.242568 19439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:00.242823 19439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:00.242893 19439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:00.246982 19439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:00.247042 19439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:00.247144 19439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:12:00.252425 19439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:40055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:00.253636 19439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:00.255179 19439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:00.265271 19444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:00.265275 19445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:00.265444 19439 server_base.cc:1034] running on GCE node
W20260924 20:12:00.265357 19447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:00.265954 19439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:00.266566 19439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:00.267726 19439 hybrid_clock.cc:648] HybridClock initialized: now 1790280720267710 us; error 32 us; skew 500 ppm
I20260924 20:12:00.269557 19439 webserver.cc:460] Webserver started at http://127.0.115.131:38019/ using document root <none> and password file <none>
I20260924 20:12:00.270150 19439 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:00.270211 19439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:00.270488 19439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:00.272637 19439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "e46cd0c22c724ac9bffbc39c432dc555"
format_stamp: "Formatted at 2026-09-24 20:12:00 on dist-test-slave-2pm7"
server_key: "868680d7b758992e8db71341b69a1773"
server_key_iv: "9e56dffcabfdbaa2bcab3aa88f90c6ab"
server_key_version: "encryptionkey@0"
I20260924 20:12:00.273196 19439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "e46cd0c22c724ac9bffbc39c432dc555"
format_stamp: "Formatted at 2026-09-24 20:12:00 on dist-test-slave-2pm7"
server_key: "868680d7b758992e8db71341b69a1773"
server_key_iv: "9e56dffcabfdbaa2bcab3aa88f90c6ab"
server_key_version: "encryptionkey@0"
I20260924 20:12:00.276880 19439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:12:00.279181 19453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:00.280602 19439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:12:00.280750 19439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "e46cd0c22c724ac9bffbc39c432dc555"
format_stamp: "Formatted at 2026-09-24 20:12:00 on dist-test-slave-2pm7"
server_key: "868680d7b758992e8db71341b69a1773"
server_key_iv: "9e56dffcabfdbaa2bcab3aa88f90c6ab"
server_key_version: "encryptionkey@0"
I20260924 20:12:00.280866 19439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:00.304859 19439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:00.305567 19439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:00.305732 19439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:00.306299 19439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:00.307322 19439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:00.307369 19439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:00.307444 19439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:00.307478 19439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:00.316665 19439 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44863
I20260924 20:12:00.316689 19566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44863 every 8 connection(s)
I20260924 20:12:00.317641 19439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:12:00.326527 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19439
I20260924 20:12:00.326646 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:12:00.326992 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.319591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:37983 (local address 127.0.115.190:40055)
0924 20:12:00.319749 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.319754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.320908 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 20:12:00.322240 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.322243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.322292 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.322368 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:00.323239 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.323774 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.324960 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.325181 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.325265 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.326395 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:00.326402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:00.326407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:00.326423 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:00.326462 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:00.326474 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:00.326565 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.326769 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.326837 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:12:00.327003 462 external_mini_cluster.cc:1428] Setting key acacaafd9d72b304a79d396b9cb03d59
I20260924 20:12:00.327942 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.320078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:40055 (local address 127.0.115.131:37983)
0924 20:12:00.320743 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.320784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:00.322030 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:00.322536 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:00.322543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:00.322623 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:00.323110 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.323123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.323883 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.323885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.324830 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.324840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.325314 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.325317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.325454 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.326094 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:12:00.326096 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:00.326137 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:12:00.326220 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:00.326571 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:00.326578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:00.326582 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:12:00.326747 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 20:12:00.327000 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260924 20:12:00.329159 19567 heartbeater.cc:347] Connected to a master server at 127.0.115.190:40055
I20260924 20:12:00.329376 19567 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:00.329890 19567 heartbeater.cc:511] Master 127.0.115.190:40055 requested a full tablet report, sending...
I20260924 20:12:00.330896 19116 ts_manager.cc:194] Registered new tserver with Master: e46cd0c22c724ac9bffbc39c432dc555 (127.0.115.131:44863)
I20260924 20:12:00.331460 19116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.131:37983
I20260924 20:12:00.341203 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:12:00.347276 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.343237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.115.190:40055)
0924 20:12:00.343389 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.343395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.343563 (+ 168us) server_negotiation.cc:407] Connection header received
0924 20:12:00.343715 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.343718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.343763 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.343859 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:00.344167 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.344695 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.345561 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.345798 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.345914 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.346707 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:00.346711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:00.346713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:00.346727 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:00.346755 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:00.346764 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:00.346893 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.347062 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.347104 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:12:00.350026 19116 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:12:00.352949 19116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:12:00.367597 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.361901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.115.130:37383)
0924 20:12:00.362977 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.362989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.363016 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:12:00.363098 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.363106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.363293 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.363454 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:00.364039 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.364774 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.366313 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.366875 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.367007 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.367221 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.367321 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.367425 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":146,"threads_started":1}
I20260924 20:12:00.367595 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.361600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:37383 (local address 127.0.0.1:42036)
0924 20:12:00.362483 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.362540 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:00.362679 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:00.363465 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:00.363471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:00.363494 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:00.363742 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.363752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.364892 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.364897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.366186 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.366196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.367018 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.367021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.367127 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.367164 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:12:00.367291 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 20:12:00.367421 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":31,"spinlock_wait_cycles":5504,"thread_start_us":89,"threads_started":1}
I20260924 20:12:00.368350 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.361901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:41293 (local address 127.0.0.1:45526)
0924 20:12:00.362482 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.362540 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:00.362679 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:00.363476 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:00.363480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:00.363517 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:00.363742 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.363752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.365248 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.365251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.366510 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.366519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.367581 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.367585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.368057 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.368083 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:12:00.368133 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 20:12:00.368205 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":143,"spinlock_wait_cycles":6656,"thread_start_us":114,"threads_started":1}
I20260924 20:12:00.368587 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.361746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.0.115.131:44863)
0924 20:12:00.363100 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.363112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.363132 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:12:00.363196 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.363200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.363339 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.363474 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:00.363922 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.365122 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.367098 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.367891 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.368073 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.368266 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.368316 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.368422 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":128,"threads_started":1}
I20260924 20:12:00.368523 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.361600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44863 (local address 127.0.0.1:47006)
0924 20:12:00.362890 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.362902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:00.362974 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:00.363614 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:00.363617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:00.363631 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:00.363756 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.363765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.365248 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.365251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.366959 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.366968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.368104 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.368108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.368192 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.368216 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:12:00.368281 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:12:00.368349 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":619,"thread_start_us":89,"threads_started":1}
I20260924 20:12:00.371088 19370 tablet_service.cc:1467] Processing CreateTablet for tablet e5b0b271b67e47a2ac8bb0994c93ca0a (DEFAULT_TABLE table=test-table [id=72cfda50c3a04ef392dea1516ed3679a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:00.371577 19501 tablet_service.cc:1467] Processing CreateTablet for tablet e5b0b271b67e47a2ac8bb0994c93ca0a (DEFAULT_TABLE table=test-table [id=72cfda50c3a04ef392dea1516ed3679a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:00.372337 19370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b0b271b67e47a2ac8bb0994c93ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:00.372599 19501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b0b271b67e47a2ac8bb0994c93ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:00.373179 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.362478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.0.115.129:41293)
0924 20:12:00.362983 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.362997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.363020 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:12:00.363103 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.363110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.363313 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.363652 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:00.363876 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.365101 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.366642 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.367432 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.372402 (+ 4970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.372603 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.372865 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.373001 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":138,"threads_started":1}
I20260924 20:12:00.375654 19238 tablet_service.cc:1467] Processing CreateTablet for tablet e5b0b271b67e47a2ac8bb0994c93ca0a (DEFAULT_TABLE table=test-table [id=72cfda50c3a04ef392dea1516ed3679a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:00.376611 19238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b0b271b67e47a2ac8bb0994c93ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:00.380300 19585 tablet_bootstrap.cc:492] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Bootstrap starting.
I20260924 20:12:00.380637 19584 tablet_bootstrap.cc:492] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Bootstrap starting.
I20260924 20:12:00.383105 19585 tablet_bootstrap.cc:654] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:00.383432 19586 tablet_bootstrap.cc:492] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: Bootstrap starting.
I20260924 20:12:00.384433 19585 log.cc:824] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:00.385756 19586 tablet_bootstrap.cc:654] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:00.386085 19584 tablet_bootstrap.cc:654] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:00.386479 19585 tablet_bootstrap.cc:492] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: No bootstrap required, opened a new log
I20260924 20:12:00.386618 19586 log.cc:824] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:00.386685 19585 ts_tablet_manager.cc:1397] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:12:00.386842 19584 log.cc:824] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:00.388711 19584 tablet_bootstrap.cc:492] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: No bootstrap required, opened a new log
I20260924 20:12:00.388717 19586 tablet_bootstrap.cc:492] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: No bootstrap required, opened a new log
I20260924 20:12:00.388907 19584 ts_tablet_manager.cc:1397] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 20:12:00.388907 19586 ts_tablet_manager.cc:1397] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:12:00.390275 19585 raft_consensus.cc:348] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.390523 19585 raft_consensus.cc:374] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:00.390576 19585 raft_consensus.cc:729] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1766c1031f264eb880e2778bf7b59af7, State: Initialized, Role: FOLLOWER
I20260924 20:12:00.391042 19585 consensus_queue.cc:260] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.392112 19585 ts_tablet_manager.cc:1428] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260924 20:12:00.392494 19436 heartbeater.cc:503] Master 127.0.115.190:40055 was elected leader, sending a full tablet report...
I20260924 20:12:00.393237 19586 raft_consensus.cc:348] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.393237 19584 raft_consensus.cc:348] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.393492 19584 raft_consensus.cc:374] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:00.393501 19586 raft_consensus.cc:374] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:00.393548 19586 raft_consensus.cc:729] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498bb557de6243558a000a9433d371cf, State: Initialized, Role: FOLLOWER
I20260924 20:12:00.393554 19584 raft_consensus.cc:729] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e46cd0c22c724ac9bffbc39c432dc555, State: Initialized, Role: FOLLOWER
I20260924 20:12:00.393996 19586 consensus_queue.cc:260] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.394002 19584 consensus_queue.cc:260] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.394663 19567 heartbeater.cc:503] Master 127.0.115.190:40055 was elected leader, sending a full tablet report...
I20260924 20:12:00.394968 19304 heartbeater.cc:503] Master 127.0.115.190:40055 was elected leader, sending a full tablet report...
I20260924 20:12:00.394984 19586 ts_tablet_manager.cc:1428] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 20:12:00.396083 19584 ts_tablet_manager.cc:1428] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260924 20:12:00.441020 19305 tablet.cc:2369] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:12:00.568778 19568 tablet.cc:2369] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:12:00.623651 19437 tablet.cc:2369] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:12:00.719123 19591 raft_consensus.cc:482] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:12:00.719403 19591 raft_consensus.cc:504] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.720683 19591 leader_election.cc:288] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1766c1031f264eb880e2778bf7b59af7 (127.0.115.130:37383), 498bb557de6243558a000a9433d371cf (127.0.115.129:41293)
I20260924 20:12:00.724397 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.720871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:37383 (local address 127.0.115.131:38413)
0924 20:12:00.721037 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.721053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:00.721154 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:00.721461 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:00.721464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:00.721481 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:00.721599 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.721605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.722335 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.722340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.723233 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.723242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.724068 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.724071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.724160 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.724182 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:12:00.724245 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 20:12:00.724295 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 20:12:00.724483 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.721099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:38413 (local address 127.0.115.130:37383)
0924 20:12:00.721234 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.721240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.721256 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:12:00.721313 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.721315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.721354 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.721446 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:00.721724 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.722220 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.723357 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.723886 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.724017 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.724068 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.724291 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.724358 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:12:00.725092 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.720988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:41293 (local address 127.0.115.131:52829)
0924 20:12:00.721387 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.721398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:00.721542 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:00.721801 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:00.721804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:00.721820 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:00.721941 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.721947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.722655 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.722659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.723613 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:00.723622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.724672 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.724677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:00.724791 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.724813 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:12:00.724882 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:12:00.724929 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":123,"threads_started":1}
I20260924 20:12:00.725108 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.721081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:52829 (local address 127.0.115.129:41293)
0924 20:12:00.721211 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.721215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.721491 (+ 276us) server_negotiation.cc:407] Connection header received
0924 20:12:00.721648 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.721652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.721702 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.721789 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:00.722059 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.722541 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.723817 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.724413 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.724535 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.724586 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.724928 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.725010 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:12:00.725304 19390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b0b271b67e47a2ac8bb0994c93ca0a" candidate_uuid: "e46cd0c22c724ac9bffbc39c432dc555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1766c1031f264eb880e2778bf7b59af7" is_pre_election: true
I20260924 20:12:00.725679 19390 raft_consensus.cc:2454] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e46cd0c22c724ac9bffbc39c432dc555 in term 0.
I20260924 20:12:00.725821 19258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b0b271b67e47a2ac8bb0994c93ca0a" candidate_uuid: "e46cd0c22c724ac9bffbc39c432dc555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498bb557de6243558a000a9433d371cf" is_pre_election: true
I20260924 20:12:00.726096 19258 raft_consensus.cc:2454] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e46cd0c22c724ac9bffbc39c432dc555 in term 0.
I20260924 20:12:00.726198 19454 leader_election.cc:302] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1766c1031f264eb880e2778bf7b59af7, e46cd0c22c724ac9bffbc39c432dc555; no voters:
I20260924 20:12:00.726543 19591 raft_consensus.cc:2781] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:12:00.726634 19591 raft_consensus.cc:482] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:12:00.726683 19591 raft_consensus.cc:3037] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:00.727671 19591 raft_consensus.cc:504] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.728031 19591 leader_election.cc:288] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [CANDIDATE]: Term 1 election: Requested vote from peers 1766c1031f264eb880e2778bf7b59af7 (127.0.115.130:37383), 498bb557de6243558a000a9433d371cf (127.0.115.129:41293)
I20260924 20:12:00.728340 19390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b0b271b67e47a2ac8bb0994c93ca0a" candidate_uuid: "e46cd0c22c724ac9bffbc39c432dc555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1766c1031f264eb880e2778bf7b59af7"
I20260924 20:12:00.728351 19258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b0b271b67e47a2ac8bb0994c93ca0a" candidate_uuid: "e46cd0c22c724ac9bffbc39c432dc555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498bb557de6243558a000a9433d371cf"
I20260924 20:12:00.728477 19390 raft_consensus.cc:3037] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:00.728513 19258 raft_consensus.cc:3037] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:00.729460 19390 raft_consensus.cc:2454] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e46cd0c22c724ac9bffbc39c432dc555 in term 1.
I20260924 20:12:00.729813 19454 leader_election.cc:302] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1766c1031f264eb880e2778bf7b59af7, e46cd0c22c724ac9bffbc39c432dc555; no voters:
I20260924 20:12:00.729557 19258 raft_consensus.cc:2454] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e46cd0c22c724ac9bffbc39c432dc555 in term 1.
I20260924 20:12:00.730012 19591 raft_consensus.cc:2781] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:00.730274 19591 raft_consensus.cc:686] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 1 LEADER]: Becoming Leader. State: Replica: e46cd0c22c724ac9bffbc39c432dc555, State: Running, Role: LEADER
I20260924 20:12:00.730613 19591 consensus_queue.cc:237] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } }
I20260924 20:12:00.733731 19115 catalog_manager.cc:5495] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 reported cstate change: term changed from 0 to 1, leader changed from <none> to e46cd0c22c724ac9bffbc39c432dc555 (127.0.115.131). New cstate: current_term: 1 leader_uuid: "e46cd0c22c724ac9bffbc39c432dc555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e46cd0c22c724ac9bffbc39c432dc555" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 } health_report { overall_health: UNKNOWN } } }
I20260924 20:12:00.820843 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.817146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.0.115.131:44863)
0924 20:12:00.817315 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:00.817321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:00.817337 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:12:00.817421 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:00.817423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:00.817479 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:00.817613 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:00.817896 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.818396 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.819297 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:00.819525 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:00.819643 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:00.819998 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:00.820416 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:00.820590 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:00.820654 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 20:12:00.820687 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 20:12:00.827139 19390 raft_consensus.cc:1261] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 1 FOLLOWER]: Refusing update from remote peer e46cd0c22c724ac9bffbc39c432dc555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:00.827149 19258 raft_consensus.cc:1261] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 1 FOLLOWER]: Refusing update from remote peer e46cd0c22c724ac9bffbc39c432dc555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:00.827852 19594 consensus_queue.cc:1035] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498bb557de6243558a000a9433d371cf" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:00.828083 19591 consensus_queue.cc:1035] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1766c1031f264eb880e2778bf7b59af7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:00.835263 19597 mvcc.cc:205] Tried to move back new op lower bound from 7332989832500015104 to 7332989832115179520. Current Snapshot: MvccSnapshot[applied={T|T < 7332989832500015104}]
I20260924 20:12:00.836525 19599 mvcc.cc:205] Tried to move back new op lower bound from 7332989832500015104 to 7332989832115179520. Current Snapshot: MvccSnapshot[applied={T|T < 7332989832500015104}]
I20260924 20:12:00.838356 19598 mvcc.cc:205] Tried to move back new op lower bound from 7332989832500015104 to 7332989832115179520. Current Snapshot: MvccSnapshot[applied={T|T < 7332989832500015104}]
I20260924 20:12:00.844750 19115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:12:00.844954 19115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:12:00.847532 19115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a0ef8092334826b7eaa392dab01b4a: Sending DeleteTablet for 3 replicas of tablet e5b0b271b67e47a2ac8bb0994c93ca0a
I20260924 20:12:00.848434 19370 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b0b271b67e47a2ac8bb0994c93ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:00 UTC) from {username='slave'} at 127.0.0.1:42036
I20260924 20:12:00.848425 19501 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b0b271b67e47a2ac8bb0994c93ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:00 UTC) from {username='slave'} at 127.0.0.1:47006
I20260924 20:12:00.848973 19606 tablet_replica.cc:330] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: stopping tablet replica
I20260924 20:12:00.848959 19238 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b0b271b67e47a2ac8bb0994c93ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:00 UTC) from {username='slave'} at 127.0.0.1:45526
I20260924 20:12:00.849407 19608 tablet_replica.cc:330] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf: stopping tablet replica
I20260924 20:12:00.849424 19606 raft_consensus.cc:2229] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:12:00.849702 19608 raft_consensus.cc:2229] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:12:00.849803 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19176
I20260924 20:12:00.849990 19606 raft_consensus.cc:2258] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:12:00.850008 19608 raft_consensus.cc:2258] T e5b0b271b67e47a2ac8bb0994c93ca0a P 498bb557de6243558a000a9433d371cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:12:00.851646 19606 ts_tablet_manager.cc:1905] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:12:00.854513 19606 ts_tablet_manager.cc:1918] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:12:00.854630 19606 log.cc:1196] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/e5b0b271b67e47a2ac8bb0994c93ca0a
I20260924 20:12:00.854926 19606 ts_tablet_manager.cc:1939] T e5b0b271b67e47a2ac8bb0994c93ca0a P 1766c1031f264eb880e2778bf7b59af7: Deleting consensus metadata
I20260924 20:12:00.856997 19607 tablet_replica.cc:330] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: stopping tablet replica
I20260924 20:12:00.857234 19100 catalog_manager.cc:4841] TS 1766c1031f264eb880e2778bf7b59af7 (127.0.115.130:37383): tablet e5b0b271b67e47a2ac8bb0994c93ca0a (table test-table [id=72cfda50c3a04ef392dea1516ed3679a]) successfully deleted
I20260924 20:12:00.857438 19607 raft_consensus.cc:2229] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:12:00.857800 19607 raft_consensus.cc:2258] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:12:00.858639 19102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:41293 (error 108)
I20260924 20:12:00.858803 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19308
I20260924 20:12:00.860898 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:00.860454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:41293 (local address 127.0.0.1:45542)
0924 20:12:00.860696 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:00.860778 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260924 20:12:00.861073 19102 catalog_manager.cc:4584] TS 498bb557de6243558a000a9433d371cf (127.0.115.129:41293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5b0b271b67e47a2ac8bb0994c93ca0a: Network error: Client connection negotiation failed: client connection to 127.0.115.129:41293: connect: Connection refused (error 111)
I20260924 20:12:00.861366 19607 ts_tablet_manager.cc:1905] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:12:00.865197 19607 ts_tablet_manager.cc:1918] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:12:00.865309 19607 log.cc:1196] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/e5b0b271b67e47a2ac8bb0994c93ca0a
I20260924 20:12:00.865636 19607 ts_tablet_manager.cc:1939] T e5b0b271b67e47a2ac8bb0994c93ca0a P e46cd0c22c724ac9bffbc39c432dc555: Deleting consensus metadata
I20260924 20:12:00.867009 19101 catalog_manager.cc:4841] TS e46cd0c22c724ac9bffbc39c432dc555 (127.0.115.131:44863): tablet e5b0b271b67e47a2ac8bb0994c93ca0a (table test-table [id=72cfda50c3a04ef392dea1516ed3679a]) successfully deleted
I20260924 20:12:00.868232 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19439
I20260924 20:12:00.873617 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19085
2026-09-24T20:12:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1415 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:00 dist-test-slave-2pm7 krb5kdc[19612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:00 dist-test-slave-2pm7 krb5kdc[19612](info): set up 2 sockets
Sep 24 20:12:00 dist-test-slave-2pm7 krb5kdc[19612](info): commencing operation
krb5kdc: starting...
W20260924 20:12:01.916240 462 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:02.043572 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35989
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:02.158337 19625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:02.158696 19625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:02.158784 19625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:02.163790 19625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:02.164039 19625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:02.164134 19625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:02.164175 19625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:02.164213 19625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:02.170328 19625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:02.171504 19625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:02.172616 19625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:02.181418 19630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:02.181422 19633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:02.181418 19631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:02.181809 19625 server_base.cc:1034] running on GCE node
I20260924 20:12:02.182457 19625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:02.183667 19625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:02.184857 19625 hybrid_clock.cc:648] HybridClock initialized: now 1790280722184836 us; error 39 us; skew 500 ppm
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:02.187631 19625 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:02.188567 19625 webserver.cc:460] Webserver started at http://127.0.115.190:38441/ using document root <none> and password file <none>
I20260924 20:12:02.189144 19625 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:02.189239 19625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:02.189476 19625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:02.191592 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "0e6e85a07d2e4f269cf7660824268f15"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "42abbbb541a630ed8d7e896ec9997a67"
server_key_iv: "713f5ea5cc51ae6f20d65bcb88c78909"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.192199 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "0e6e85a07d2e4f269cf7660824268f15"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "42abbbb541a630ed8d7e896ec9997a67"
server_key_iv: "713f5ea5cc51ae6f20d65bcb88c78909"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.196094 19625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:12:02.198488 19640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.199687 19625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:02.199831 19625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "0e6e85a07d2e4f269cf7660824268f15"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "42abbbb541a630ed8d7e896ec9997a67"
server_key_iv: "713f5ea5cc51ae6f20d65bcb88c78909"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.199949 19625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:02.215497 19625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:02.218869 19625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:02.219092 19625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:02.226667 19625 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:35989
I20260924 20:12:02.226681 19692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:35989 every 8 connection(s)
I20260924 20:12:02.227625 19625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:02.230427 19693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:02.230485 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19625
I20260924 20:12:02.230645 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:02.230962 462 external_mini_cluster.cc:1428] Setting key 6881919f6b8c1ac7a754a344e3b3504d
I20260924 20:12:02.238009 19693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15: Bootstrap starting.
I20260924 20:12:02.240589 19693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15: Neither blocks nor log segments found. Creating new log.
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:02.241459 19693 log.cc:824] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:02.244050 19693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15: No bootstrap required, opened a new log
I20260924 20:12:02.246927 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:02.232428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.0.115.190:35989)
0924 20:12:02.232901 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:02.232920 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:02.232976 (+ 56us) server_negotiation.cc:407] Connection header received
0924 20:12:02.233985 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:02.234013 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:02.234481 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:02.234865 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:02.235537 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.237133 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.238026 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.238715 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.238873 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:02.241479 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:02.241486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:02.241505 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:02.241540 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:02.243534 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:02.243998 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:02.244004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:02.244010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:02.244163 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:02.244440 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:02.244445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:02.244449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:02.244970 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:02.245157 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:02.245517 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 20:12:02.245830 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":131,"threads_started":1}
I20260924 20:12:02.248677 19693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6e85a07d2e4f269cf7660824268f15" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35989 } }
I20260924 20:12:02.248924 19693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:02.248984 19693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6e85a07d2e4f269cf7660824268f15, State: Initialized, Role: FOLLOWER
I20260924 20:12:02.249436 19693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6e85a07d2e4f269cf7660824268f15" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35989 } }
I20260924 20:12:02.249558 19693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:02.249599 19693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:02.249671 19693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:02.250524 19693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6e85a07d2e4f269cf7660824268f15" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35989 } }
I20260924 20:12:02.250869 19693 leader_election.cc:302] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e6e85a07d2e4f269cf7660824268f15; no voters:
I20260924 20:12:02.251228 19693 leader_election.cc:288] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:02.251293 19698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:02.251526 19698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [term 1 LEADER]: Becoming Leader. State: Replica: 0e6e85a07d2e4f269cf7660824268f15, State: Running, Role: LEADER
I20260924 20:12:02.251853 19698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6e85a07d2e4f269cf7660824268f15" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35989 } }
I20260924 20:12:02.252574 19693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:02.253674 19699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e6e85a07d2e4f269cf7660824268f15. Latest consensus state: current_term: 1 leader_uuid: "0e6e85a07d2e4f269cf7660824268f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6e85a07d2e4f269cf7660824268f15" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35989 } } }
I20260924 20:12:02.253719 19700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e6e85a07d2e4f269cf7660824268f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6e85a07d2e4f269cf7660824268f15" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35989 } } }
I20260924 20:12:02.253808 19699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:02.253815 19700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:02.254181 19707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:02.257331 19707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:02.263564 19707 catalog_manager.cc:1347] Generated new cluster ID: 31c30a06ec144e79b6abb70696ffc434
I20260924 20:12:02.263643 19707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:02.275434 19707 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:02.276778 19707 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:02.283440 19707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15: Generated new TSK 0
I20260924 20:12:02.284600 19707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.115.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
Entry for principal kudu/127.0.115.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab.
I20260924 20:12:02.316169 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35989
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:02.436334 19719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:02.436646 19719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:02.436728 19719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:02.440878 19719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:02.440969 19719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:02.441056 19719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:12:02.446568 19719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.129.keytab
--principal=kudu/127.0.115.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:02.447762 19719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:02.448918 19719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:02.459327 19725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:02.459327 19727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:02.459532 19719 server_base.cc:1034] running on GCE node
W20260924 20:12:02.459455 19724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:02.460088 19719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:02.460813 19719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:02.461992 19719 hybrid_clock.cc:648] HybridClock initialized: now 1790280722461969 us; error 38 us; skew 500 ppm
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:02.465361 19719 init.cc:377] Logged in from keytab as kudu/127.0.115.129@KRBTEST.COM (short username kudu)
I20260924 20:12:02.466352 19719 webserver.cc:460] Webserver started at http://127.0.115.129:42215/ using document root <none> and password file <none>
I20260924 20:12:02.467232 19719 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:02.467338 19719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:02.467644 19719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:02.470782 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "a76320df1efa45889d1febabc870e0a7"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "838acd8532b70e231ad7ae096f333944"
server_key_iv: "04ee642a62dc54e33ab209df205c01b2"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.471648 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "a76320df1efa45889d1febabc870e0a7"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "838acd8532b70e231ad7ae096f333944"
server_key_iv: "04ee642a62dc54e33ab209df205c01b2"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.476972 19719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 20:12:02.480808 19734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.482563 19719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:12:02.482697 19719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "a76320df1efa45889d1febabc870e0a7"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "838acd8532b70e231ad7ae096f333944"
server_key_iv: "04ee642a62dc54e33ab209df205c01b2"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.482832 19719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:02.526686 19719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:02.529930 19719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:02.530092 19719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:02.530650 19719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:02.531458 19719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:02.531517 19719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.531560 19719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:02.531584 19719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.541322 19719 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:38509
I20260924 20:12:02.541343 19847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:38509 every 8 connection(s)
I20260924 20:12:02.542286 19719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:12:02.544122 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19719
I20260924 20:12:02.544252 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:12:02.544596 462 external_mini_cluster.cc:1428] Setting key a9a0e7af189d240930fd84234519136e
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.129@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.130@KRBTEST.COM; defaulting to no policy
I20260924 20:12:02.557658 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:02.544256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:43741 (local address 127.0.115.190:35989)
0924 20:12:02.544427 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:02.544431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:02.545532 (+ 1101us) server_negotiation.cc:407] Connection header received
0924 20:12:02.546656 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:02.546659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:02.546709 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:02.546815 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:02.548274 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.548811 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.550596 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.550822 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.550900 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:02.553985 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:02.553988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:02.553991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:02.554015 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:02.555584 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:02.556196 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:02.556199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:02.556201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:02.556286 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:02.556620 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:02.556622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:02.556623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:02.556806 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:02.556879 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:02.557403 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 20:12:02.557524 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.130@KRBTEST.COM" created.
I20260924 20:12:02.558712 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:02.544798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35989 (local address 127.0.115.129:43741)
0924 20:12:02.545358 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:02.545423 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:02.546444 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:02.547005 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:02.547012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:02.547380 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:02.548101 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:02.548126 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.548935 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.548941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:02.550472 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:02.550489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.550985 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.550992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:02.551196 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:02.551801 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:02.551817 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:02.553797 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:02.555703 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:02.555708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:02.555722 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:02.556096 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:02.556372 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:02.556374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:02.556375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:02.556532 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:02.556893 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:02.556898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:02.557156 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 20:12:02.557312 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 20:12:02.557606 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":141,"threads_started":1}
I20260924 20:12:02.560117 19848 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35989
I20260924 20:12:02.560382 19848 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:02.560899 19848 heartbeater.cc:511] Master 127.0.115.190:35989 requested a full tablet report, sending...
I20260924 20:12:02.562529 19655 ts_manager.cc:194] Registered new tserver with Master: a76320df1efa45889d1febabc870e0a7 (127.0.115.129:38509)
I20260924 20:12:02.563669 19655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.129@KRBTEST.COM'} at 127.0.115.129:43741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
Entry for principal kudu/127.0.115.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab.
I20260924 20:12:02.573416 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35989
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:02.687412 19853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:02.687731 19853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:02.687808 19853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:02.692019 19853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:02.692113 19853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:02.692204 19853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:12:02.697592 19853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.130.keytab
--principal=kudu/127.0.115.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:02.698787 19853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:02.699838 19853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:02.709878 19859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:02.709919 19861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:02.709911 19858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:02.710057 19853 server_base.cc:1034] running on GCE node
I20260924 20:12:02.710490 19853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:02.711310 19853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:02.712520 19853 hybrid_clock.cc:648] HybridClock initialized: now 1790280722712499 us; error 29 us; skew 500 ppm
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:02.715237 19853 init.cc:377] Logged in from keytab as kudu/127.0.115.130@KRBTEST.COM (short username kudu)
I20260924 20:12:02.716153 19853 webserver.cc:460] Webserver started at http://127.0.115.130:33541/ using document root <none> and password file <none>
I20260924 20:12:02.716722 19853 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:02.716813 19853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:02.717051 19853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:02.719144 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "3738c29cd1bf432fad6879957e4bebc4"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "755ab474495d9879fd07ac8966741dbe"
server_key_iv: "4f56e2e1b37a57d24591b47efbaa743e"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.719740 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "3738c29cd1bf432fad6879957e4bebc4"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "755ab474495d9879fd07ac8966741dbe"
server_key_iv: "4f56e2e1b37a57d24591b47efbaa743e"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.723460 19853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:12:02.725947 19868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.727173 19853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:02.727306 19853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "3738c29cd1bf432fad6879957e4bebc4"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "755ab474495d9879fd07ac8966741dbe"
server_key_iv: "4f56e2e1b37a57d24591b47efbaa743e"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.727418 19853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:02.745673 19853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:02.748870 19853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:02.749080 19853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:02.749752 19853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:02.750607 19853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:02.750675 19853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.750746 19853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:02.750787 19853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.760304 19853 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:37797
I20260924 20:12:02.760325 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:37797 every 8 connection(s)
I20260924 20:12:02.761235 19853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:12:02.770166 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19853
I20260924 20:12:02.770273 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.130@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:02.770655 462 external_mini_cluster.cc:1428] Setting key 5f709e5e6377b253d72d86a34c5e3794
I20260924 20:12:02.774696 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:02.763115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35055 (local address 127.0.115.190:35989)
0924 20:12:02.763252 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:02.763256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:02.764146 (+ 890us) server_negotiation.cc:407] Connection header received
0924 20:12:02.765365 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:02.765368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:02.765413 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:02.765488 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:02.766613 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.767139 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.768305 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.768542 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.768621 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:02.771108 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:02.771112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:02.771114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:02.771148 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:02.772608 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:02.773138 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:02.773146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:02.773148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:02.773220 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:02.773564 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:02.773568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:02.773570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:02.773754 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:02.773860 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:02.774389 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 20:12:02.774527 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:12:02.775516 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:02.763497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35989 (local address 127.0.115.130:35055)
0924 20:12:02.763992 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:02.764048 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:02.765180 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:02.765649 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:02.765656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:02.766012 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:02.766490 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:02.766503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.767244 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.767246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:02.768204 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:02.768214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:02.768669 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:02.768671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:02.768807 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:02.769353 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:02.769369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:02.770904 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:02.772731 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:02.772737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:02.772753 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:02.773029 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:02.773313 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:02.773317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:02.773319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:02.773465 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:02.773861 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:02.773869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:02.774127 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 20:12:02.774333 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 20:12:02.774675 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260924 20:12:02.776700 19982 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35989
I20260924 20:12:02.776957 19982 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:02.777451 19982 heartbeater.cc:511] Master 127.0.115.190:35989 requested a full tablet report, sending...
I20260924 20:12:02.778537 19655 ts_manager.cc:194] Registered new tserver with Master: 3738c29cd1bf432fad6879957e4bebc4 (127.0.115.130:37797)
I20260924 20:12:02.779102 19655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.130@KRBTEST.COM'} at 127.0.115.130:35055
WARNING: no policy specified for kudu/127.0.115.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
Entry for principal kudu/127.0.115.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab.
I20260924 20:12:02.799286 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35989
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:02.912876 19987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:02.913154 19987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:02.913236 19987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:02.917346 19987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:02.917403 19987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:02.917528 19987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:12:02.923003 19987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/krb5kdc/kudu_127.0.115.131.keytab
--principal=kudu/127.0.115.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:02.924167 19987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:02.925231 19987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:02.935143 19992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:02.935160 19993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:02.935148 19995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:02.935575 19987 server_base.cc:1034] running on GCE node
I20260924 20:12:02.935930 19987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:02.936609 19987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:02.937772 19987 hybrid_clock.cc:648] HybridClock initialized: now 1790280722937751 us; error 28 us; skew 500 ppm
Sep 24 20:12:02 dist-test-slave-2pm7 krb5kdc[19612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:02.940503 19987 init.cc:377] Logged in from keytab as kudu/127.0.115.131@KRBTEST.COM (short username kudu)
I20260924 20:12:02.941319 19987 webserver.cc:460] Webserver started at http://127.0.115.131:45061/ using document root <none> and password file <none>
I20260924 20:12:02.941831 19987 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:02.941887 19987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:02.942062 19987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:02.944087 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "5eef00e212a9454ca089317d15c5c004"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "be3955f8e5768ef08fb0da3875b59437"
server_key_iv: "9f90e13758666e8fc68272c490850097"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.944591 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "5eef00e212a9454ca089317d15c5c004"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "be3955f8e5768ef08fb0da3875b59437"
server_key_iv: "9f90e13758666e8fc68272c490850097"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.948235 19987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:12:02.950703 20002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.952044 19987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:02.952186 19987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "5eef00e212a9454ca089317d15c5c004"
format_stamp: "Formatted at 2026-09-24 20:12:02 on dist-test-slave-2pm7"
server_key: "be3955f8e5768ef08fb0da3875b59437"
server_key_iv: "9f90e13758666e8fc68272c490850097"
server_key_version: "encryptionkey@0"
I20260924 20:12:02.952318 19987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:02.994468 19987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:02.997485 19987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:02.997696 19987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:02.998313 19987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:02.999171 19987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:02.999241 19987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:02.999313 19987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:02.999364 19987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:03.008473 19987 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:42499
I20260924 20:12:03.008494 20115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:42499 every 8 connection(s)
I20260924 20:12:03.009370 19987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:12:03.016228 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 19987
I20260924 20:12:03.016384 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:12:03.016667 462 external_mini_cluster.cc:1428] Setting key 94137fd2cf5ca4daa59af0125f9fbe1d
Sep 24 20:12:03 dist-test-slave-2pm7 krb5kdc[19612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.131@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:03.022802 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.011215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:53735 (local address 127.0.115.190:35989)
0924 20:12:03.011367 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.011372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.012254 (+ 882us) server_negotiation.cc:407] Connection header received
0924 20:12:03.013107 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.013110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.013158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.013231 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:03.014356 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.014884 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.016616 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.016894 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.017012 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.019733 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:03.019736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:03.019738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:03.019763 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:03.021075 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:03.021531 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:03.021534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:03.021535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:03.021606 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:03.021896 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:03.021898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:03.021899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:03.022067 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:03.022140 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.022586 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.022680 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:12:03.023625 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.011558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35989 (local address 127.0.115.131:53735)
0924 20:12:03.012073 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.012131 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:03.012905 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:03.013394 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:03.013401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:03.013757 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:03.014224 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.014237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.015101 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.015104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.016482 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.016495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.017055 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.017060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.017237 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.017884 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:03.017900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:03.019589 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:03.021177 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:03.021182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:03.021195 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:03.021446 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:03.021684 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:03.021687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:03.021688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:03.021817 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:03.022151 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:03.022156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:03.022385 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 20:12:03.022540 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 20:12:03.022786 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260924 20:12:03.024789 20116 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35989
I20260924 20:12:03.025050 20116 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:03.025571 20116 heartbeater.cc:511] Master 127.0.115.190:35989 requested a full tablet report, sending...
I20260924 20:12:03.026624 19655 ts_manager.cc:194] Registered new tserver with Master: 5eef00e212a9454ca089317d15c5c004 (127.0.115.131:42499)
I20260924 20:12:03.027197 19655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.115.131@KRBTEST.COM'} at 127.0.115.131:53735
I20260924 20:12:03.030578 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:12:03.039700 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.032656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.115.190:35989)
0924 20:12:03.032804 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.032809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.032994 (+ 185us) server_negotiation.cc:407] Connection header received
0924 20:12:03.033079 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.033081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.033121 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.033184 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:03.033838 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.034346 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.035273 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.035482 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.035574 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.036876 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:03.036880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:03.036882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:03.036903 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:03.038298 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:03.038650 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:03.038653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:03.038654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:03.038723 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:03.038938 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:03.038939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:03.038941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:03.039139 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:03.039238 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.039409 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.039496 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:12:03.042563 19655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:12:03.045786 19655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:12:03.060575 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.054451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:37797 (local address 127.0.0.1:50462)
0924 20:12:03.054845 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.054930 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:03.055075 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:03.056031 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:03.056041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:03.056069 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:03.056230 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.056241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.057707 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.057710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.058903 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.058917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.059974 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.059978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.060108 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.060147 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 20:12:03.060267 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 20:12:03.060413 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":4,"thread_start_us":94,"threads_started":1}
I20260924 20:12:03.060613 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.054756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.0.115.130:37797)
0924 20:12:03.055418 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.055430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.055455 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:12:03.055541 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.055546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.055762 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.056012 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:03.056353 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.057579 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.059045 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.059818 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.059970 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.060140 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.060338 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.060453 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":162,"threads_started":1}
I20260924 20:12:03.061048 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.055055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:42499 (local address 127.0.0.1:44624)
0924 20:12:03.055460 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.055474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:03.055557 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:03.056537 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:03.056540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:03.056558 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:03.056657 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.056664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.058061 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.058065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.059280 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.059290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.060626 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.060632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.060732 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.060760 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:12:03.060831 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:12:03.060900 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":28,"thread_start_us":95,"threads_started":1}
I20260924 20:12:03.061456 20131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.055467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.0.115.131:42499)
0924 20:12:03.055879 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.055897 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.055922 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:12:03.056032 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.056037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.056231 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.056381 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:03.056770 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.057913 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.059423 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.060318 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.060411 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.060555 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.061226 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.061326 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":90,"threads_started":1}
I20260924 20:12:03.061926 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.055210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:38509 (local address 127.0.0.1:54612)
0924 20:12:03.056854 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.056878 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:03.056985 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:03.057387 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:03.057390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:03.057409 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:03.057497 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.057502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.058808 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.058811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.060149 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.060162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.061528 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.061531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.061617 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.061639 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:12:03.061734 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 20:12:03.061808 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"thread_start_us":89,"threads_started":1}
I20260924 20:12:03.062858 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.056020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.0.115.129:38509)
0924 20:12:03.056702 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.056719 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.056986 (+ 267us) server_negotiation.cc:407] Connection header received
0924 20:12:03.057083 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.057088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.057254 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.057389 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:03.057621 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.058658 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.060290 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.061398 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.061526 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.061658 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.062624 (+ 966us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.062723 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":106,"threads_started":1}
I20260924 20:12:03.064231 19916 tablet_service.cc:1467] Processing CreateTablet for tablet 4d968f76046746ecb2a61eb449dd9ffd (DEFAULT_TABLE table=test-table [id=ad66387c94234e8d8dc252438a6e822a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:03.064388 20050 tablet_service.cc:1467] Processing CreateTablet for tablet 4d968f76046746ecb2a61eb449dd9ffd (DEFAULT_TABLE table=test-table [id=ad66387c94234e8d8dc252438a6e822a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:03.065303 19916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d968f76046746ecb2a61eb449dd9ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:03.065443 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d968f76046746ecb2a61eb449dd9ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:03.065645 19782 tablet_service.cc:1467] Processing CreateTablet for tablet 4d968f76046746ecb2a61eb449dd9ffd (DEFAULT_TABLE table=test-table [id=ad66387c94234e8d8dc252438a6e822a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:03.066668 19782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d968f76046746ecb2a61eb449dd9ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:03.072222 20133 tablet_bootstrap.cc:492] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Bootstrap starting.
I20260924 20:12:03.073174 20134 tablet_bootstrap.cc:492] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Bootstrap starting.
I20260924 20:12:03.074446 20135 tablet_bootstrap.cc:492] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: Bootstrap starting.
I20260924 20:12:03.074934 20133 tablet_bootstrap.cc:654] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:03.075363 20134 tablet_bootstrap.cc:654] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:03.075928 20133 log.cc:824] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:03.076478 20134 log.cc:824] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:03.076782 20135 tablet_bootstrap.cc:654] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:03.077678 20135 log.cc:824] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:03.078292 20134 tablet_bootstrap.cc:492] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: No bootstrap required, opened a new log
I20260924 20:12:03.078292 20133 tablet_bootstrap.cc:492] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: No bootstrap required, opened a new log
I20260924 20:12:03.078492 20133 ts_tablet_manager.cc:1397] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 20:12:03.078492 20134 ts_tablet_manager.cc:1397] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:12:03.079342 20135 tablet_bootstrap.cc:492] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: No bootstrap required, opened a new log
I20260924 20:12:03.079484 20135 ts_tablet_manager.cc:1397] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:12:03.082624 20133 raft_consensus.cc:348] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.082624 20134 raft_consensus.cc:348] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.082933 20133 raft_consensus.cc:374] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:03.082932 20134 raft_consensus.cc:374] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:03.082767 20135 raft_consensus.cc:348] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.082993 20134 raft_consensus.cc:729] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eef00e212a9454ca089317d15c5c004, State: Initialized, Role: FOLLOWER
I20260924 20:12:03.083037 20133 raft_consensus.cc:729] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3738c29cd1bf432fad6879957e4bebc4, State: Initialized, Role: FOLLOWER
I20260924 20:12:03.083077 20135 raft_consensus.cc:374] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:03.083156 20135 raft_consensus.cc:729] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a76320df1efa45889d1febabc870e0a7, State: Initialized, Role: FOLLOWER
I20260924 20:12:03.083487 20134 consensus_queue.cc:260] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.083487 20133 consensus_queue.cc:260] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.083645 20135 consensus_queue.cc:260] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.084604 20133 ts_tablet_manager.cc:1428] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 20:12:03.084604 20134 ts_tablet_manager.cc:1428] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:12:03.084668 20135 ts_tablet_manager.cc:1428] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:12:03.084981 19982 heartbeater.cc:503] Master 127.0.115.190:35989 was elected leader, sending a full tablet report...
I20260924 20:12:03.085099 19848 heartbeater.cc:503] Master 127.0.115.190:35989 was elected leader, sending a full tablet report...
I20260924 20:12:03.085983 20116 heartbeater.cc:503] Master 127.0.115.190:35989 was elected leader, sending a full tablet report...
W20260924 20:12:03.260362 20117 tablet.cc:2369] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:12:03.262367 19983 tablet.cc:2369] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:12:03.277822 20141 raft_consensus.cc:482] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:12:03.278105 20141 raft_consensus.cc:504] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.279264 20141 leader_election.cc:288] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3738c29cd1bf432fad6879957e4bebc4 (127.0.115.130:37797), 5eef00e212a9454ca089317d15c5c004 (127.0.115.131:42499)
I20260924 20:12:03.284895 20131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.279804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:58537 (local address 127.0.115.131:42499)
0924 20:12:03.280051 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.280055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.280171 (+ 116us) server_negotiation.cc:407] Connection header received
0924 20:12:03.280343 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.280347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.280401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.280491 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:03.280892 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.281667 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.283490 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.284151 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.284263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.284320 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.284684 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.284758 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 20:12:03.284895 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.279674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:39725 (local address 127.0.115.130:37797)
0924 20:12:03.279868 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.279873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.280177 (+ 304us) server_negotiation.cc:407] Connection header received
0924 20:12:03.280344 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.280347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.280405 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.280554 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:03.280881 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.281669 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.283490 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.284136 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.284263 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.284320 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.284684 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.284758 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:12:03.284977 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.279596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:37797 (local address 127.0.115.129:39725)
0924 20:12:03.280058 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.280073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:03.280225 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:03.280556 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:03.280558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:03.280569 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:03.280760 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.280767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.281790 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.281794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.283364 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.283370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.284268 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.284271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.284516 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.284570 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 20:12:03.284652 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:12:03.284752 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":309,"spinlock_wait_cycles":13440,"thread_start_us":130,"threads_started":1}
I20260924 20:12:03.284977 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.279596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:42499 (local address 127.0.115.129:58537)
0924 20:12:03.280058 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.280073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:03.280225 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:03.280528 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:03.280531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:03.280553 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:03.280760 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.280767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.281790 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.281794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.283356 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.283370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.284268 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.284271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.284518 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.284577 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 20:12:03.284652 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:12:03.284752 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":316,"spinlock_wait_cycles":33408,"thread_start_us":90,"threads_started":1}
I20260924 20:12:03.285881 20070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d968f76046746ecb2a61eb449dd9ffd" candidate_uuid: "a76320df1efa45889d1febabc870e0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eef00e212a9454ca089317d15c5c004" is_pre_election: true
I20260924 20:12:03.286075 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d968f76046746ecb2a61eb449dd9ffd" candidate_uuid: "a76320df1efa45889d1febabc870e0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3738c29cd1bf432fad6879957e4bebc4" is_pre_election: true
I20260924 20:12:03.286235 20070 raft_consensus.cc:2454] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a76320df1efa45889d1febabc870e0a7 in term 0.
I20260924 20:12:03.286382 19936 raft_consensus.cc:2454] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a76320df1efa45889d1febabc870e0a7 in term 0.
I20260924 20:12:03.286803 19737 leader_election.cc:302] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eef00e212a9454ca089317d15c5c004, a76320df1efa45889d1febabc870e0a7; no voters:
I20260924 20:12:03.287122 20141 raft_consensus.cc:2781] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:12:03.287232 20141 raft_consensus.cc:482] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:12:03.287292 20141 raft_consensus.cc:3037] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:03.288558 20141 raft_consensus.cc:504] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
I20260924 20:12:03.288975 20141 leader_election.cc:288] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 3738c29cd1bf432fad6879957e4bebc4 (127.0.115.130:37797), 5eef00e212a9454ca089317d15c5c004 (127.0.115.131:42499)
I20260924 20:12:03.289310 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d968f76046746ecb2a61eb449dd9ffd" candidate_uuid: "a76320df1efa45889d1febabc870e0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3738c29cd1bf432fad6879957e4bebc4"
I20260924 20:12:03.289325 20070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d968f76046746ecb2a61eb449dd9ffd" candidate_uuid: "a76320df1efa45889d1febabc870e0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eef00e212a9454ca089317d15c5c004"
I20260924 20:12:03.289455 19936 raft_consensus.cc:3037] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:03.289469 20070 raft_consensus.cc:3037] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:03.290495 20070 raft_consensus.cc:2454] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a76320df1efa45889d1febabc870e0a7 in term 1.
I20260924 20:12:03.290542 19936 raft_consensus.cc:2454] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a76320df1efa45889d1febabc870e0a7 in term 1.
I20260924 20:12:03.290854 19737 leader_election.cc:302] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eef00e212a9454ca089317d15c5c004, a76320df1efa45889d1febabc870e0a7; no voters:
I20260924 20:12:03.291028 20141 raft_consensus.cc:2781] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:03.291272 20141 raft_consensus.cc:686] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 1 LEADER]: Becoming Leader. State: Replica: a76320df1efa45889d1febabc870e0a7, State: Running, Role: LEADER
I20260924 20:12:03.291623 20141 consensus_queue.cc:237] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } }
W20260924 20:12:03.293608 19849 tablet.cc:2369] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:12:03.294888 19656 catalog_manager.cc:5495] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a76320df1efa45889d1febabc870e0a7 (127.0.115.129). New cstate: current_term: 1 leader_uuid: "a76320df1efa45889d1febabc870e0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a76320df1efa45889d1febabc870e0a7" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 38509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 } health_report { overall_health: UNKNOWN } } }
I20260924 20:12:03.355000 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.351274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.0.115.129:38509)
0924 20:12:03.351429 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.351433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.351448 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:12:03.351558 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.351560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.351616 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.351708 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:03.351962 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.352554 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.353473 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.353707 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.353814 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.354113 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:03.354601 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:03.354694 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.354753 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.354809 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:12:03.361364 20070 raft_consensus.cc:1261] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 1 FOLLOWER]: Refusing update from remote peer a76320df1efa45889d1febabc870e0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:03.361366 19936 raft_consensus.cc:1261] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 1 FOLLOWER]: Refusing update from remote peer a76320df1efa45889d1febabc870e0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:03.362119 20141 consensus_queue.cc:1035] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eef00e212a9454ca089317d15c5c004" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:03.362375 20144 consensus_queue.cc:1035] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3738c29cd1bf432fad6879957e4bebc4" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:03.368947 20147 mvcc.cc:205] Tried to move back new op lower bound from 7332989842880368640 to 7332989842605182976. Current Snapshot: MvccSnapshot[applied={T|T < 7332989842880368640}]
I20260924 20:12:03.370340 20149 mvcc.cc:205] Tried to move back new op lower bound from 7332989842880368640 to 7332989842605182976. Current Snapshot: MvccSnapshot[applied={T|T < 7332989842880368640}]
I20260924 20:12:03.382071 19657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:12:03.382236 19657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:12:03.384938 19657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e6e85a07d2e4f269cf7660824268f15: Sending DeleteTablet for 3 replicas of tablet 4d968f76046746ecb2a61eb449dd9ffd
I20260924 20:12:03.385851 19782 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d968f76046746ecb2a61eb449dd9ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:03 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:54612
I20260924 20:12:03.385850 19916 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d968f76046746ecb2a61eb449dd9ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:03 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:50462
I20260924 20:12:03.385892 20050 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d968f76046746ecb2a61eb449dd9ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:03 UTC) from {username='kudu', principal='kudu/127.0.115.190@KRBTEST.COM'} at 127.0.0.1:44624
I20260924 20:12:03.386411 20159 tablet_replica.cc:330] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: stopping tablet replica
I20260924 20:12:03.386439 20157 tablet_replica.cc:330] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7: stopping tablet replica
I20260924 20:12:03.386766 20159 raft_consensus.cc:2229] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:12:03.386766 20157 raft_consensus.cc:2229] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:12:03.387125 20159 raft_consensus.cc:2258] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:12:03.387187 20157 raft_consensus.cc:2258] T 4d968f76046746ecb2a61eb449dd9ffd P a76320df1efa45889d1febabc870e0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:12:03.387264 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19719
I20260924 20:12:03.387565 20158 tablet_replica.cc:330] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: stopping tablet replica
I20260924 20:12:03.387852 20158 raft_consensus.cc:2229] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:12:03.388168 20158 raft_consensus.cc:2258] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:12:03.389467 20159 ts_tablet_manager.cc:1905] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:12:03.390916 20158 ts_tablet_manager.cc:1905] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:12:03.392411 20159 ts_tablet_manager.cc:1918] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:12:03.392519 20159 log.cc:1196] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/4d968f76046746ecb2a61eb449dd9ffd
I20260924 20:12:03.392915 20159 ts_tablet_manager.cc:1939] T 4d968f76046746ecb2a61eb449dd9ffd P 5eef00e212a9454ca089317d15c5c004: Deleting consensus metadata
I20260924 20:12:03.393707 20158 ts_tablet_manager.cc:1918] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:12:03.393805 20158 log.cc:1196] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/4d968f76046746ecb2a61eb449dd9ffd
I20260924 20:12:03.394142 20158 ts_tablet_manager.cc:1939] T 4d968f76046746ecb2a61eb449dd9ffd P 3738c29cd1bf432fad6879957e4bebc4: Deleting consensus metadata
I20260924 20:12:03.394378 19643 catalog_manager.cc:4841] TS 5eef00e212a9454ca089317d15c5c004 (127.0.115.131:42499): tablet 4d968f76046746ecb2a61eb449dd9ffd (table test-table [id=ad66387c94234e8d8dc252438a6e822a]) successfully deleted
W20260924 20:12:03.395135 19642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:38509 (error 108)
I20260924 20:12:03.395368 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19853
I20260924 20:12:03.396751 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.396195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:38509 (local address 127.0.0.1:54630)
0924 20:12:03.396551 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.396638 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:38509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
W20260924 20:12:03.396947 19642 catalog_manager.cc:4584] TS a76320df1efa45889d1febabc870e0a7 (127.0.115.129:38509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d968f76046746ecb2a61eb449dd9ffd: Network error: Client connection negotiation failed: client connection to 127.0.115.129:38509: connect: Connection refused (error 111)
I20260924 20:12:03.401515 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19987
I20260924 20:12:03.402856 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.402475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:37797 (local address 127.0.0.1:50476)
0924 20:12:03.402683 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.402748 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.130:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 20:12:03.407445 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 19625
2026-09-24T20:12:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2535 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T20:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:03Z Disabled control of system clock
I20260924 20:12:03.438367 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35363
--webserver_interface=127.0.115.190
--webserver_port=0
--builtin_ntp_servers=127.0.115.148:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:12:03.553506 20164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:03.553782 20164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:03.553819 20164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:03.557775 20164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:03.557830 20164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:03.557847 20164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:03.557863 20164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:03.557878 20164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:03.562568 20164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:35363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:03.563642 20164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:03.565013 20164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:03.573642 20169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:03.573652 20170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:03.573652 20172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:03.573977 20164 server_base.cc:1034] running on GCE node
I20260924 20:12:03.574594 20164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:03.575732 20164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:03.576926 20164 hybrid_clock.cc:648] HybridClock initialized: now 1790280723576926 us; error 47 us; skew 500 ppm
I20260924 20:12:03.582499 20164 webserver.cc:460] Webserver started at http://127.0.115.190:43669/ using document root <none> and password file <none>
I20260924 20:12:03.583191 20164 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:03.583294 20164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:03.583544 20164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:03.585798 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "ed912b2b2c3548be97a376e94b8c9d6d"
format_stamp: "Formatted at 2026-09-24 20:12:03 on dist-test-slave-2pm7"
server_key: "6be0e0eed83a6c1797d78692e6e69eb5"
server_key_iv: "188460c71f612acbc7055dc1f50bd589"
server_key_version: "encryptionkey@0"
I20260924 20:12:03.586402 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "ed912b2b2c3548be97a376e94b8c9d6d"
format_stamp: "Formatted at 2026-09-24 20:12:03 on dist-test-slave-2pm7"
server_key: "6be0e0eed83a6c1797d78692e6e69eb5"
server_key_iv: "188460c71f612acbc7055dc1f50bd589"
server_key_version: "encryptionkey@0"
I20260924 20:12:03.590267 20164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:12:03.593667 20178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:03.594980 20164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:12:03.595134 20164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "ed912b2b2c3548be97a376e94b8c9d6d"
format_stamp: "Formatted at 2026-09-24 20:12:03 on dist-test-slave-2pm7"
server_key: "6be0e0eed83a6c1797d78692e6e69eb5"
server_key_iv: "188460c71f612acbc7055dc1f50bd589"
server_key_version: "encryptionkey@0"
I20260924 20:12:03.595255 20164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:03.621814 20164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:03.622617 20164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:03.622839 20164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:03.630407 20164 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:35363
I20260924 20:12:03.630492 20230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:35363 every 8 connection(s)
I20260924 20:12:03.631455 20164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:03.634687 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 20164
I20260924 20:12:03.634748 20231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:03.634835 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:03.635133 462 external_mini_cluster.cc:1428] Setting key 41cacac4f210463dbdfdacb8ccccb49f
I20260924 20:12:03.642716 20231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d: Bootstrap starting.
I20260924 20:12:03.645515 20231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:03.646281 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.636823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.0.115.190:35363)
0924 20:12:03.637528 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.637549 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.637598 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:12:03.638677 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.638703 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.639252 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.639638 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:03.640133 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.641636 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.642484 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.643168 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.643325 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.644197 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:03.644203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:03.644219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:03.644710 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:03.644781 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:03.644808 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:03.644919 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.645016 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.645288 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260924 20:12:03.646448 20231 log.cc:824] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:03.648635 20231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d: No bootstrap required, opened a new log
I20260924 20:12:03.652298 20231 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35363 } }
I20260924 20:12:03.652529 20231 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:03.652563 20231 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed912b2b2c3548be97a376e94b8c9d6d, State: Initialized, Role: FOLLOWER
I20260924 20:12:03.653025 20231 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35363 } }
I20260924 20:12:03.653175 20231 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:03.653261 20231 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:03.653378 20231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:03.654331 20231 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35363 } }
I20260924 20:12:03.654681 20231 leader_election.cc:302] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed912b2b2c3548be97a376e94b8c9d6d; no voters:
I20260924 20:12:03.655004 20231 leader_election.cc:288] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:03.655140 20236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:03.655417 20236 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [term 1 LEADER]: Becoming Leader. State: Replica: ed912b2b2c3548be97a376e94b8c9d6d, State: Running, Role: LEADER
I20260924 20:12:03.655738 20236 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35363 } }
I20260924 20:12:03.656317 20231 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:03.657583 20238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed912b2b2c3548be97a376e94b8c9d6d. Latest consensus state: current_term: 1 leader_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35363 } } }
I20260924 20:12:03.657707 20238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:03.658080 20237 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed912b2b2c3548be97a376e94b8c9d6d" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 35363 } } }
I20260924 20:12:03.658145 20245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:03.658205 20237 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:03.661434 20245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:03.666651 20245 catalog_manager.cc:1347] Generated new cluster ID: 485775e12c4d4eec96698fb2fadf8eaf
I20260924 20:12:03.666738 20245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:03.696702 20245 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:03.697892 20245 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:03.705667 20245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d: Generated new TSK 0
I20260924 20:12:03.706379 20245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:03.711639 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.129:0
--local_ip_for_outbound_sockets=127.0.115.129
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35363
--builtin_ntp_servers=127.0.115.148:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:12:03.825484 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:03.825803 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:03.825872 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:03.830099 20255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:03.830180 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:03.830273 20255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.129
I20260924 20:12:03.835196 20255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.115.129
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.129
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:03.836549 20255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:03.838040 20255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:03.848057 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:03.848070 20255 server_base.cc:1034] running on GCE node
W20260924 20:12:03.848050 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:03.848050 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:03.848730 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:03.849438 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:03.850637 20255 hybrid_clock.cc:648] HybridClock initialized: now 1790280723850621 us; error 28 us; skew 500 ppm
I20260924 20:12:03.852629 20255 webserver.cc:460] Webserver started at http://127.0.115.129:38221/ using document root <none> and password file <none>
I20260924 20:12:03.853232 20255 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:03.853328 20255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:03.853576 20255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:03.855660 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data/instance:
uuid: "edee237e336d4d328270ac6a076a8b5c"
format_stamp: "Formatted at 2026-09-24 20:12:03 on dist-test-slave-2pm7"
server_key: "c8d2f134d40ea535cd9cea48d8016169"
server_key_iv: "2ee83e6eaab09582c01faadb9286b173"
server_key_version: "encryptionkey@0"
I20260924 20:12:03.856307 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/wal/instance:
uuid: "edee237e336d4d328270ac6a076a8b5c"
format_stamp: "Formatted at 2026-09-24 20:12:03 on dist-test-slave-2pm7"
server_key: "c8d2f134d40ea535cd9cea48d8016169"
server_key_iv: "2ee83e6eaab09582c01faadb9286b173"
server_key_version: "encryptionkey@0"
I20260924 20:12:03.860186 20255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:12:03.862618 20269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:03.864028 20255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:03.864163 20255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/wal
uuid: "edee237e336d4d328270ac6a076a8b5c"
format_stamp: "Formatted at 2026-09-24 20:12:03 on dist-test-slave-2pm7"
server_key: "c8d2f134d40ea535cd9cea48d8016169"
server_key_iv: "2ee83e6eaab09582c01faadb9286b173"
server_key_version: "encryptionkey@0"
I20260924 20:12:03.864286 20255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:03.880838 20255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:03.881614 20255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:03.881829 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:03.882452 20255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:03.883435 20255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:03.883525 20255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:03.883620 20255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:03.883674 20255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:03.892923 20255 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.129:46135
I20260924 20:12:03.892947 20382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.129:46135 every 8 connection(s)
I20260924 20:12:03.893882 20255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/data/info.pb
I20260924 20:12:03.898476 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 20255
I20260924 20:12:03.898627 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-0/wal/instance
I20260924 20:12:03.898989 462 external_mini_cluster.cc:1428] Setting key e2f8db1efe248f1fe7b6c062f22b4b43
I20260924 20:12:03.901227 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.130:0
--local_ip_for_outbound_sockets=127.0.115.130
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35363
--builtin_ntp_servers=127.0.115.148:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:12:03.904683 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.895828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:50709 (local address 127.0.115.190:35363)
0924 20:12:03.896009 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:03.896014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:03.897139 (+ 1125us) server_negotiation.cc:407] Connection header received
0924 20:12:03.898927 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:03.898930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:03.899005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:03.899107 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:03.900324 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.901093 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.902469 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.902727 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.902812 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.904201 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:03.904204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:03.904206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:03.904219 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:03.904241 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:03.904249 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:03.904306 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:03.904504 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 20:12:03.904567 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:12:03.906018 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:03.896283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35363 (local address 127.0.115.129:50709)
0924 20:12:03.896950 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:03.896991 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:03.898651 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:03.899410 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:03.899424 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:03.899563 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:03.900173 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.900190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.901233 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.901236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.902310 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:03.902327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:03.902895 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:03.902901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:03.903137 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:03.903844 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:12:03.903849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:03.903906 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:12:03.904028 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:03.904335 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:03.904347 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:03.904356 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:12:03.904483 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 20:12:03.904820 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260924 20:12:03.907451 20383 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35363
I20260924 20:12:03.907745 20383 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:03.908385 20383 heartbeater.cc:511] Master 127.0.115.190:35363 requested a full tablet report, sending...
I20260924 20:12:03.910091 20195 ts_manager.cc:194] Registered new tserver with Master: edee237e336d4d328270ac6a076a8b5c (127.0.115.129:46135)
I20260924 20:12:03.911396 20195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.129:50709
W20260924 20:12:04.014194 20386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:04.014482 20386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:04.014557 20386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:04.018525 20386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:04.018576 20386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:04.018630 20386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.130
I20260924 20:12:04.023320 20386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.115.130
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.130
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:04.024446 20386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:04.025841 20386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:04.035705 20392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:04.035737 20391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:04.035744 20394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:04.035970 20386 server_base.cc:1034] running on GCE node
I20260924 20:12:04.036474 20386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:04.037155 20386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:04.038311 20386 hybrid_clock.cc:648] HybridClock initialized: now 1790280724038276 us; error 43 us; skew 500 ppm
I20260924 20:12:04.040098 20386 webserver.cc:460] Webserver started at http://127.0.115.130:35077/ using document root <none> and password file <none>
I20260924 20:12:04.040671 20386 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:04.040732 20386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:04.040974 20386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:04.043018 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data/instance:
uuid: "d6b388f173a84f5082d56481946c9bb7"
format_stamp: "Formatted at 2026-09-24 20:12:04 on dist-test-slave-2pm7"
server_key: "98153d6d36b507a2300c56242a447117"
server_key_iv: "0d92c6829ba005fd13e4eef6d37a5259"
server_key_version: "encryptionkey@0"
I20260924 20:12:04.043599 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal/instance:
uuid: "d6b388f173a84f5082d56481946c9bb7"
format_stamp: "Formatted at 2026-09-24 20:12:04 on dist-test-slave-2pm7"
server_key: "98153d6d36b507a2300c56242a447117"
server_key_iv: "0d92c6829ba005fd13e4eef6d37a5259"
server_key_version: "encryptionkey@0"
I20260924 20:12:04.047855 20386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260924 20:12:04.050596 20400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:04.051878 20386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:04.052064 20386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal
uuid: "d6b388f173a84f5082d56481946c9bb7"
format_stamp: "Formatted at 2026-09-24 20:12:04 on dist-test-slave-2pm7"
server_key: "98153d6d36b507a2300c56242a447117"
server_key_iv: "0d92c6829ba005fd13e4eef6d37a5259"
server_key_version: "encryptionkey@0"
I20260924 20:12:04.052176 20386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:04.065804 20386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:04.066522 20386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:04.066720 20386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:04.067390 20386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:04.068503 20386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:04.068584 20386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:04.068637 20386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:04.068670 20386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:04.077210 20386 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.130:38573
I20260924 20:12:04.077231 20513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.130:38573 every 8 connection(s)
I20260924 20:12:04.078509 20386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/data/info.pb
I20260924 20:12:04.087512 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.080393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.130:35663 (local address 127.0.115.190:35363)
0924 20:12:04.080568 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.080573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.081683 (+ 1110us) server_negotiation.cc:407] Connection header received
0924 20:12:04.083022 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.083024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.083072 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.083146 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:04.084128 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.084665 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.085771 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.085982 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.086053 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.087092 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:04.087095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:04.087097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:04.087110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:04.087130 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:04.087138 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:04.087194 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.087337 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.087389 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 20:12:04.088294 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 20386
I20260924 20:12:04.088420 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal/instance
I20260924 20:12:04.088593 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.080847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35363 (local address 127.0.115.130:35663)
0924 20:12:04.081479 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.081564 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:04.082829 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:04.083309 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:04.083316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:04.083401 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:04.083877 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.083889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.084777 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.084779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.085666 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.085675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.086117 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.086120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.086262 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.086837 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:12:04.086839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:04.086879 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:12:04.086942 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:04.087211 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:04.087218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:04.087222 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:12:04.087316 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 20:12:04.087536 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260924 20:12:04.088784 462 external_mini_cluster.cc:1428] Setting key b23f17471c9f2d881a267c0e006e5b3d
I20260924 20:12:04.089852 20514 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35363
I20260924 20:12:04.090085 20514 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:04.090602 20514 heartbeater.cc:511] Master 127.0.115.190:35363 requested a full tablet report, sending...
I20260924 20:12:04.090673 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.115.131:0
--local_ip_for_outbound_sockets=127.0.115.131
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35363
--builtin_ntp_servers=127.0.115.148:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:12:04.091730 20195 ts_manager.cc:194] Registered new tserver with Master: d6b388f173a84f5082d56481946c9bb7 (127.0.115.130:38573)
I20260924 20:12:04.092391 20195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.130:35663
W20260924 20:12:04.204697 20517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:04.204955 20517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:04.204991 20517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:04.209322 20517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:04.209378 20517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:04.209458 20517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.115.131
I20260924 20:12:04.214217 20517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.115.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.115.131
--webserver_port=0
--tserver_master_addrs=127.0.115.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.115.131
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:04.215341 20517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:04.216903 20517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:04.226616 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:04.226619 20522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:04.226621 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:04.226732 20517 server_base.cc:1034] running on GCE node
I20260924 20:12:04.227336 20517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:04.228045 20517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:04.229205 20517 hybrid_clock.cc:648] HybridClock initialized: now 1790280724229175 us; error 48 us; skew 500 ppm
I20260924 20:12:04.230935 20517 webserver.cc:460] Webserver started at http://127.0.115.131:45043/ using document root <none> and password file <none>
I20260924 20:12:04.231519 20517 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:04.231580 20517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:04.231813 20517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:04.233947 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data/instance:
uuid: "5862591dd17243e8a75787f55bc3c699"
format_stamp: "Formatted at 2026-09-24 20:12:04 on dist-test-slave-2pm7"
server_key: "bc255f878043349179b4754c0cb291cf"
server_key_iv: "63a8648225460b8c5f42d3b50954af0c"
server_key_version: "encryptionkey@0"
I20260924 20:12:04.234552 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal/instance:
uuid: "5862591dd17243e8a75787f55bc3c699"
format_stamp: "Formatted at 2026-09-24 20:12:04 on dist-test-slave-2pm7"
server_key: "bc255f878043349179b4754c0cb291cf"
server_key_iv: "63a8648225460b8c5f42d3b50954af0c"
server_key_version: "encryptionkey@0"
I20260924 20:12:04.238262 20517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:12:04.240715 20531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:04.241933 20517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:12:04.242069 20517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal
uuid: "5862591dd17243e8a75787f55bc3c699"
format_stamp: "Formatted at 2026-09-24 20:12:04 on dist-test-slave-2pm7"
server_key: "bc255f878043349179b4754c0cb291cf"
server_key_iv: "63a8648225460b8c5f42d3b50954af0c"
server_key_version: "encryptionkey@0"
I20260924 20:12:04.242183 20517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:04.252776 20517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:04.253435 20517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:04.253623 20517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:04.254159 20517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:12:04.255074 20517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:12:04.255142 20517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:04.255214 20517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:12:04.255267 20517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:04.264055 20517 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.131:44635
I20260924 20:12:04.264084 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.131:44635 every 8 connection(s)
I20260924 20:12:04.264958 20517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/data/info.pb
I20260924 20:12:04.267635 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 20517
I20260924 20:12:04.267772 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal/instance
I20260924 20:12:04.268119 462 external_mini_cluster.cc:1428] Setting key 960f75adaa691ebb539e5f662698bbe5
I20260924 20:12:04.274353 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.266795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.131:60857 (local address 127.0.115.190:35363)
0924 20:12:04.266938 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.266942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.268217 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 20:12:04.269957 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.269972 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.270032 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.270134 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:04.271002 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.271518 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.272670 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.272882 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.272966 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.273933 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:04.273936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:04.273938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:04.273949 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:04.273976 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:04.273983 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:04.274039 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.274181 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.274229 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:12:04.275218 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.267193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:35363 (local address 127.0.115.131:60857)
0924 20:12:04.268066 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.268107 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:04.269761 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:04.270309 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:04.270316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:04.270397 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:04.270872 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.270884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.271620 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.271623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.272568 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.272577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.273007 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.273010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.273154 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.273697 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:12:04.273699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:04.273737 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:12:04.273798 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:04.274061 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:04.274067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:04.274072 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:12:04.274164 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 20:12:04.274390 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":179,"threads_started":1}
I20260924 20:12:04.276371 20645 heartbeater.cc:347] Connected to a master server at 127.0.115.190:35363
I20260924 20:12:04.276614 20645 heartbeater.cc:464] Registering TS with master...
I20260924 20:12:04.277125 20645 heartbeater.cc:511] Master 127.0.115.190:35363 requested a full tablet report, sending...
I20260924 20:12:04.278147 20195 ts_manager.cc:194] Registered new tserver with Master: 5862591dd17243e8a75787f55bc3c699 (127.0.115.131:44635)
I20260924 20:12:04.278757 20195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.115.131:60857
I20260924 20:12:04.281869 462 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:12:04.287479 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.283737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.115.190:35363)
0924 20:12:04.283877 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.283882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.284121 (+ 239us) server_negotiation.cc:407] Connection header received
0924 20:12:04.284276 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.284283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.284320 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.284396 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:04.284689 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.285187 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.286040 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.286253 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.286358 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.287060 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:04.287063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:04.287066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:04.287083 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:04.287099 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:04.287107 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:04.287179 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.287294 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.287345 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:12:04.290060 20195 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:12:04.292990 20195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:12:04.307085 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.301046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44635 (local address 127.0.0.1:34618)
0924 20:12:04.301488 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.301541 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:04.301658 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:04.302930 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:04.302933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:04.302951 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:04.303073 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.303080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.304454 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.304457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.305442 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.305451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.306491 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.306498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.306661 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.306700 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 20:12:04.306814 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 20:12:04.306935 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260924 20:12:04.307140 20658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.301154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.0.115.131:44635)
0924 20:12:04.302505 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.302517 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.302545 (+ 28us) server_negotiation.cc:407] Connection header received
0924 20:12:04.302605 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.302608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.302782 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.302925 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:04.303189 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.304334 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.305563 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.306352 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.306456 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.306642 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.306861 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.306975 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":83,"threads_started":1}
I20260924 20:12:04.307660 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.301788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:46135 (local address 127.0.0.1:36318)
0924 20:12:04.302958 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.302970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:04.303058 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:04.303494 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:04.303497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:04.303521 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:04.303619 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.303626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.304717 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.304723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.305786 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.305796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.306644 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.306648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.306741 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.306759 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:12:04.306852 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:12:04.307539 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":597,"thread_start_us":119,"threads_started":1}
I20260924 20:12:04.307895 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.301969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.0.115.129:46135)
0924 20:12:04.302326 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.302338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.303153 (+ 815us) server_negotiation.cc:407] Connection header received
0924 20:12:04.303240 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.303244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.303368 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.303511 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:04.303738 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.304619 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.305912 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.306448 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.306553 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.306737 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.306941 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.307034 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":95,"threads_started":1}
I20260924 20:12:04.308724 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.301129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:38573 (local address 127.0.0.1:33252)
0924 20:12:04.302046 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.302059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:04.302142 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:04.302778 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:04.302784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:04.302807 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:04.302955 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.302973 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.304503 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.304506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.307098 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.307111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.308338 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.308342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.308438 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.308459 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:12:04.308530 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:12:04.308592 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":1432,"thread_start_us":83,"threads_started":1}
I20260924 20:12:04.309160 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.301846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.0.115.130:38573)
0924 20:12:04.302289 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.302300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.302325 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:12:04.302407 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.302412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.302630 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.302754 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:04.303080 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.304334 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.307247 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.307917 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.308095 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.308276 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.308937 (+ 661us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.309043 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":118,"threads_started":1}
I20260924 20:12:04.310298 20579 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2045ecd5f8486fa3bdc6d323a1e604 (DEFAULT_TABLE table=test-table [id=8fa1531d87264a0ca6360c1f99d2e3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:04.311046 20317 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2045ecd5f8486fa3bdc6d323a1e604 (DEFAULT_TABLE table=test-table [id=8fa1531d87264a0ca6360c1f99d2e3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:04.311482 20579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2045ecd5f8486fa3bdc6d323a1e604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:04.311982 20317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2045ecd5f8486fa3bdc6d323a1e604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:04.311779 20448 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2045ecd5f8486fa3bdc6d323a1e604 (DEFAULT_TABLE table=test-table [id=8fa1531d87264a0ca6360c1f99d2e3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:12:04.312786 20448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2045ecd5f8486fa3bdc6d323a1e604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:04.318176 20662 tablet_bootstrap.cc:492] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c: Bootstrap starting.
I20260924 20:12:04.319095 20663 tablet_bootstrap.cc:492] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Bootstrap starting.
I20260924 20:12:04.320428 20664 tablet_bootstrap.cc:492] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Bootstrap starting.
I20260924 20:12:04.320766 20662 tablet_bootstrap.cc:654] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:04.321297 20663 tablet_bootstrap.cc:654] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:04.321687 20662 log.cc:824] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:04.322094 20663 log.cc:824] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:04.322292 20664 tablet_bootstrap.cc:654] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:04.323067 20664 log.cc:824] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:04.323896 20662 tablet_bootstrap.cc:492] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c: No bootstrap required, opened a new log
I20260924 20:12:04.324090 20663 tablet_bootstrap.cc:492] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: No bootstrap required, opened a new log
I20260924 20:12:04.324178 20662 ts_tablet_manager.cc:1397] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260924 20:12:04.324273 20663 ts_tablet_manager.cc:1397] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260924 20:12:04.325032 20664 tablet_bootstrap.cc:492] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: No bootstrap required, opened a new log
I20260924 20:12:04.325203 20664 ts_tablet_manager.cc:1397] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:12:04.327488 20662 raft_consensus.cc:348] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.327735 20662 raft_consensus.cc:374] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:04.327788 20662 raft_consensus.cc:729] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edee237e336d4d328270ac6a076a8b5c, State: Initialized, Role: FOLLOWER
I20260924 20:12:04.328289 20662 consensus_queue.cc:260] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.328543 20663 raft_consensus.cc:348] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.328799 20663 raft_consensus.cc:374] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:04.328853 20663 raft_consensus.cc:729] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862591dd17243e8a75787f55bc3c699, State: Initialized, Role: FOLLOWER
I20260924 20:12:04.329083 20383 heartbeater.cc:503] Master 127.0.115.190:35363 was elected leader, sending a full tablet report...
I20260924 20:12:04.329301 20662 ts_tablet_manager.cc:1428] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 20:12:04.329321 20663 consensus_queue.cc:260] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.329933 20645 heartbeater.cc:503] Master 127.0.115.190:35363 was elected leader, sending a full tablet report...
I20260924 20:12:04.330268 20663 ts_tablet_manager.cc:1428] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:12:04.329393 20664 raft_consensus.cc:348] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.330601 20664 raft_consensus.cc:374] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:04.330659 20664 raft_consensus.cc:729] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b388f173a84f5082d56481946c9bb7, State: Initialized, Role: FOLLOWER
I20260924 20:12:04.331060 20664 consensus_queue.cc:260] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.331691 20514 heartbeater.cc:503] Master 127.0.115.190:35363 was elected leader, sending a full tablet report...
I20260924 20:12:04.332114 20664 ts_tablet_manager.cc:1428] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260924 20:12:04.395150 20384 tablet.cc:2369] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:12:04.516026 20646 tablet.cc:2369] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:12:04.579757 20515 tablet.cc:2369] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:12:04.596446 20668 raft_consensus.cc:482] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:12:04.596711 20668 raft_consensus.cc:504] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.598031 20668 leader_election.cc:288] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5862591dd17243e8a75787f55bc3c699 (127.0.115.131:44635), d6b388f173a84f5082d56481946c9bb7 (127.0.115.130:38573)
I20260924 20:12:04.601920 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.598376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:53257 (local address 127.0.115.130:38573)
0924 20:12:04.598518 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.598523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.598694 (+ 171us) server_negotiation.cc:407] Connection header received
0924 20:12:04.598885 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.598888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.598944 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.599043 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:04.599332 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.599863 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.600975 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.601362 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.601494 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.601546 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.601731 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.601794 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:12:04.601938 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.598291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.130:38573 (local address 127.0.115.129:53257)
0924 20:12:04.598600 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.598613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:04.598745 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:04.599057 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:04.599060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:04.599078 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:04.599205 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.599211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.600004 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.600007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.600841 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.600850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.601494 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.601497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.601594 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.601616 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:12:04.601713 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 20:12:04.601782 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260924 20:12:04.602528 20468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2045ecd5f8486fa3bdc6d323a1e604" candidate_uuid: "edee237e336d4d328270ac6a076a8b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b388f173a84f5082d56481946c9bb7" is_pre_election: true
I20260924 20:12:04.602764 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.598319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.131:44635 (local address 127.0.115.129:55225)
0924 20:12:04.598872 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.598884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:04.599013 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:04.599378 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:04.599382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:04.599400 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:12:04.599530 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.599536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.600561 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.600566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.601416 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:04.601426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.602296 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.602300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:04.602425 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.602443 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:12:04.602546 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 20:12:04.602616 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":74,"threads_started":1}
I20260924 20:12:04.602757 20658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.598635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.115.129:55225 (local address 127.0.115.131:44635)
0924 20:12:04.598775 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.598781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.598965 (+ 184us) server_negotiation.cc:407] Connection header received
0924 20:12:04.599191 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.599194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.599246 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.599361 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:12:04.599654 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.600431 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.601552 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.602039 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.602187 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.602232 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.602565 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.602643 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:12:04.602864 20468 raft_consensus.cc:2454] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edee237e336d4d328270ac6a076a8b5c in term 0.
I20260924 20:12:04.603339 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2045ecd5f8486fa3bdc6d323a1e604" candidate_uuid: "edee237e336d4d328270ac6a076a8b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862591dd17243e8a75787f55bc3c699" is_pre_election: true
I20260924 20:12:04.603361 20270 leader_election.cc:302] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6b388f173a84f5082d56481946c9bb7, edee237e336d4d328270ac6a076a8b5c; no voters:
I20260924 20:12:04.603574 20599 raft_consensus.cc:2454] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edee237e336d4d328270ac6a076a8b5c in term 0.
I20260924 20:12:04.603633 20668 raft_consensus.cc:2781] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:12:04.603726 20668 raft_consensus.cc:482] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:12:04.603778 20668 raft_consensus.cc:3037] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:04.604866 20668 raft_consensus.cc:504] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.605235 20668 leader_election.cc:288] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [CANDIDATE]: Term 1 election: Requested vote from peers 5862591dd17243e8a75787f55bc3c699 (127.0.115.131:44635), d6b388f173a84f5082d56481946c9bb7 (127.0.115.130:38573)
I20260924 20:12:04.605564 20468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2045ecd5f8486fa3bdc6d323a1e604" candidate_uuid: "edee237e336d4d328270ac6a076a8b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b388f173a84f5082d56481946c9bb7"
I20260924 20:12:04.605708 20468 raft_consensus.cc:3037] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:04.605705 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2045ecd5f8486fa3bdc6d323a1e604" candidate_uuid: "edee237e336d4d328270ac6a076a8b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862591dd17243e8a75787f55bc3c699"
I20260924 20:12:04.605863 20599 raft_consensus.cc:3037] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:04.606735 20468 raft_consensus.cc:2454] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edee237e336d4d328270ac6a076a8b5c in term 1.
I20260924 20:12:04.607014 20270 leader_election.cc:302] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6b388f173a84f5082d56481946c9bb7, edee237e336d4d328270ac6a076a8b5c; no voters:
I20260924 20:12:04.607036 20599 raft_consensus.cc:2454] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edee237e336d4d328270ac6a076a8b5c in term 1.
I20260924 20:12:04.607184 20668 raft_consensus.cc:2781] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:04.607426 20668 raft_consensus.cc:686] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [term 1 LEADER]: Becoming Leader. State: Replica: edee237e336d4d328270ac6a076a8b5c, State: Running, Role: LEADER
I20260924 20:12:04.607769 20668 consensus_queue.cc:237] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } }
I20260924 20:12:04.610921 20195 catalog_manager.cc:5495] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to edee237e336d4d328270ac6a076a8b5c (127.0.115.129). New cstate: current_term: 1 leader_uuid: "edee237e336d4d328270ac6a076a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edee237e336d4d328270ac6a076a8b5c" member_type: VOTER last_known_addr { host: "127.0.115.129" port: 46135 } health_report { overall_health: HEALTHY } } }
I20260924 20:12:04.669780 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.666380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.0.115.129:46135)
0924 20:12:04.666515 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:04.666519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:04.666533 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:12:04.666626 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:04.666628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:04.666685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:04.666769 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:04.667005 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.667525 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.668391 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:04.668620 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:04.668724 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:04.668989 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:04.669399 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:04.669489 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:04.669551 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 20:12:04.669597 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:12:04.675395 20599 raft_consensus.cc:1261] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 1 FOLLOWER]: Refusing update from remote peer edee237e336d4d328270ac6a076a8b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:04.675395 20468 raft_consensus.cc:1261] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 1 FOLLOWER]: Refusing update from remote peer edee237e336d4d328270ac6a076a8b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:04.676162 20673 consensus_queue.cc:1035] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5862591dd17243e8a75787f55bc3c699" member_type: VOTER last_known_addr { host: "127.0.115.131" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:04.676366 20668 consensus_queue.cc:1035] T 6c2045ecd5f8486fa3bdc6d323a1e604 P edee237e336d4d328270ac6a076a8b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b388f173a84f5082d56481946c9bb7" member_type: VOTER last_known_addr { host: "127.0.115.130" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:04.684250 20678 mvcc.cc:205] Tried to move back new op lower bound from 7332989848263303168 to 7332989847995916288. Current Snapshot: MvccSnapshot[applied={T|T < 7332989848263303168}]
I20260924 20:12:04.692741 20195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:12:04.692919 20195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:12:04.695322 20195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed912b2b2c3548be97a376e94b8c9d6d: Sending DeleteTablet for 3 replicas of tablet 6c2045ecd5f8486fa3bdc6d323a1e604
I20260924 20:12:04.696070 20448 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2045ecd5f8486fa3bdc6d323a1e604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:04 UTC) from {username='slave'} at 127.0.0.1:33252
I20260924 20:12:04.696082 20579 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2045ecd5f8486fa3bdc6d323a1e604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:04 UTC) from {username='slave'} at 127.0.0.1:34618
I20260924 20:12:04.696758 20687 tablet_replica.cc:330] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: stopping tablet replica
I20260924 20:12:04.696723 20317 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2045ecd5f8486fa3bdc6d323a1e604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:12:04 UTC) from {username='slave'} at 127.0.0.1:36318
I20260924 20:12:04.696779 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 20255
I20260924 20:12:04.697083 20687 raft_consensus.cc:2229] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:12:04.697360 20687 raft_consensus.cc:2258] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:12:04.697710 20686 tablet_replica.cc:330] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: stopping tablet replica
I20260924 20:12:04.698066 20686 raft_consensus.cc:2229] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:12:04.698354 20686 raft_consensus.cc:2258] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:12:04.699203 20687 ts_tablet_manager.cc:1905] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:12:04.700240 20686 ts_tablet_manager.cc:1905] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:12:04.702267 20686 ts_tablet_manager.cc:1918] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:12:04.702380 20686 log.cc:1196] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-2/wal/wals/6c2045ecd5f8486fa3bdc6d323a1e604
I20260924 20:12:04.702659 20686 ts_tablet_manager.cc:1939] T 6c2045ecd5f8486fa3bdc6d323a1e604 P 5862591dd17243e8a75787f55bc3c699: Deleting consensus metadata
I20260924 20:12:04.702704 20687 ts_tablet_manager.cc:1918] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:12:04.702800 20687 log.cc:1196] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Deleting WAL directory at /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790280466534371-462-0/minicluster-data/ts-1/wal/wals/6c2045ecd5f8486fa3bdc6d323a1e604
I20260924 20:12:04.703109 20687 ts_tablet_manager.cc:1939] T 6c2045ecd5f8486fa3bdc6d323a1e604 P d6b388f173a84f5082d56481946c9bb7: Deleting consensus metadata
I20260924 20:12:04.703888 20181 catalog_manager.cc:4841] TS 5862591dd17243e8a75787f55bc3c699 (127.0.115.131:44635): tablet 6c2045ecd5f8486fa3bdc6d323a1e604 (table test-table [id=8fa1531d87264a0ca6360c1f99d2e3a7]) successfully deleted
I20260924 20:12:04.704298 20179 catalog_manager.cc:4841] TS d6b388f173a84f5082d56481946c9bb7 (127.0.115.130:38573): tablet 6c2045ecd5f8486fa3bdc6d323a1e604 (table test-table [id=8fa1531d87264a0ca6360c1f99d2e3a7]) successfully deleted
W20260924 20:12:04.705755 20180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.115.129:46135 (error 108)
I20260924 20:12:04.705821 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 20386
I20260924 20:12:04.708452 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:04.708180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.129:46135 (local address 127.0.0.1:36332)
0924 20:12:04.708313 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:04.708381 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:12:04.708667 20180 catalog_manager.cc:4584] TS edee237e336d4d328270ac6a076a8b5c (127.0.115.129:46135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c2045ecd5f8486fa3bdc6d323a1e604: Network error: Client connection negotiation failed: client connection to 127.0.115.129:46135: connect: Connection refused (error 111)
I20260924 20:12:04.712173 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 20517
I20260924 20:12:04.717597 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 20164
2026-09-24T20:12:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1308 ms)
[----------] 6 tests from EncryptionPolicyTest (10866 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:04 dist-test-slave-2pm7 krb5kdc[20691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:04 dist-test-slave-2pm7 krb5kdc[20691](info): set up 2 sockets
Sep 24 20:12:04 dist-test-slave-2pm7 krb5kdc[20691](info): commencing operation
krb5kdc: starting...
W20260924 20:12:05.748278 462 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:05 dist-test-slave-2pm7 krb5kdc[20691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:05.870826 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:05.983172 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:05.983515 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:05.983601 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:05.987703 20704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:05.987818 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:05.987857 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:05.987896 20704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:05.987928 20704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:05.993417 20704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:05.994634 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:05.995743 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:06.004283 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:06.004283 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:06.004362 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:06.004485 20704 server_base.cc:1034] running on GCE node
I20260924 20:12:06.005467 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:06.006634 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:06.007875 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790280726007845 us; error 27 us; skew 500 ppm
Sep 24 20:12:06 dist-test-slave-2pm7 krb5kdc[20691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:06.010819 20704 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:06.011811 20704 webserver.cc:460] Webserver started at http://127.0.0.1:36519/ using document root <none> and password file <none>
I20260924 20:12:06.012465 20704 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:06.012579 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:06.012833 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:06.014963 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "947b276865a6467b878453690375edb4"
format_stamp: "Formatted at 2026-09-24 20:12:06 on dist-test-slave-2pm7"
server_key: "595327f1038cde389a3bf4f658f69736"
server_key_iv: "ea6277b73c11299c0be7364ad26581b6"
server_key_version: "encryptionkey@0"
I20260924 20:12:06.015555 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "947b276865a6467b878453690375edb4"
format_stamp: "Formatted at 2026-09-24 20:12:06 on dist-test-slave-2pm7"
server_key: "595327f1038cde389a3bf4f658f69736"
server_key_iv: "ea6277b73c11299c0be7364ad26581b6"
server_key_version: "encryptionkey@0"
I20260924 20:12:06.019363 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 20:12:06.021944 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:06.023245 20704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:06.023401 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "947b276865a6467b878453690375edb4"
format_stamp: "Formatted at 2026-09-24 20:12:06 on dist-test-slave-2pm7"
server_key: "595327f1038cde389a3bf4f658f69736"
server_key_iv: "ea6277b73c11299c0be7364ad26581b6"
server_key_version: "encryptionkey@0"
I20260924 20:12:06.023527 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:06.063220 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:06.066427 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:06.066651 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:06.074010 20771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34161 every 8 connection(s)
I20260924 20:12:06.074030 20704 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34161
I20260924 20:12:06.075119 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:06.077536 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 20704
I20260924 20:12:06.077682 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:06.078013 462 external_mini_cluster.cc:1428] Setting key 73790ddb29a6f412b011dedc72dcbd1c
I20260924 20:12:06.078311 20772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:06.085065 20772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4: Bootstrap starting.
Sep 24 20:12:06 dist-test-slave-2pm7 krb5kdc[20691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:12:06.087411 20772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:06.088224 20772 log.cc:824] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:06.090234 20772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4: No bootstrap required, opened a new log
I20260924 20:12:06.092382 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:06.079394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39303 (local address 127.0.0.1:34161)
0924 20:12:06.079801 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:06.079817 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:06.079854 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:12:06.080582 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:06.080600 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:06.080950 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:06.081297 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:06.082147 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:06.083166 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:06.084105 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:06.084591 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:06.084765 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:06.087366 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:06.087371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:06.087386 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:06.087414 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:06.089401 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:06.089928 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:06.089933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:06.089938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:06.090048 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:06.090393 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:06.090401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:06.090402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:06.090797 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:06.091028 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:06.091354 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 20:12:06.091584 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260924 20:12:06.093974 20772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b276865a6467b878453690375edb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34161 } }
I20260924 20:12:06.094244 20772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:06.094338 20772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947b276865a6467b878453690375edb4, State: Initialized, Role: FOLLOWER
I20260924 20:12:06.094772 20772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b276865a6467b878453690375edb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34161 } }
I20260924 20:12:06.094942 20772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:06.095019 20772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:06.095150 20772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:06.095428 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:06.096419 20772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b276865a6467b878453690375edb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34161 } }
I20260924 20:12:06.096835 20772 leader_election.cc:302] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 947b276865a6467b878453690375edb4; no voters:
I20260924 20:12:06.097204 20772 leader_election.cc:288] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:06.097306 20781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:06.097565 20781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [term 1 LEADER]: Becoming Leader. State: Replica: 947b276865a6467b878453690375edb4, State: Running, Role: LEADER
I20260924 20:12:06.097872 20781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b276865a6467b878453690375edb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34161 } }
I20260924 20:12:06.098646 20772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:06.099071 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 947b276865a6467b878453690375edb4. Latest consensus state: current_term: 1 leader_uuid: "947b276865a6467b878453690375edb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b276865a6467b878453690375edb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34161 } } }
I20260924 20:12:06.099234 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:06.099519 20782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "947b276865a6467b878453690375edb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b276865a6467b878453690375edb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34161 } } }
I20260924 20:12:06.099612 20782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:06.099687 20786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:06.103019 20786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:06.107963 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:06.098067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42913 (local address 127.0.0.1:34161)
0924 20:12:06.098174 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:06.098178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:06.098464 (+ 286us) server_negotiation.cc:407] Connection header received
0924 20:12:06.098588 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:06.098591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:06.098638 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:06.098715 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:06.100034 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:06.100856 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:06.102067 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:06.102414 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:06.102520 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:06.104345 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:06.104349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:06.104352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:06.104377 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:06.105955 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:06.106445 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:06.106448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:06.106450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:06.106534 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:06.106861 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:06.106864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:06.106866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:06.107049 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:06.107200 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:06.107476 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 20:12:06.107781 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:12:06.109391 20786 catalog_manager.cc:1347] Generated new cluster ID: 1bac6347a28a4838802d1a3ee183ee47
I20260924 20:12:06.109452 20786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:06.127818 20786 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:06.129169 20786 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:06.134629 20786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 947b276865a6467b878453690375edb4: Generated new TSK 0
I20260924 20:12:06.135463 20786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:06.143208 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 20704
W20260924 20:12:06.149310 20689 connection.cc:535] client connection to 127.0.0.1:34161 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34161): Connection reset by peer (error 104)
2026-09-24T20:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:06 dist-test-slave-2pm7 krb5kdc[20803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:06 dist-test-slave-2pm7 krb5kdc[20803](info): set up 2 sockets
Sep 24 20:12:06 dist-test-slave-2pm7 krb5kdc[20803](info): commencing operation
krb5kdc: starting...
W20260924 20:12:07.188416 462 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:07 dist-test-slave-2pm7 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:07.316228 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:07.431595 20816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:07.431881 20816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:07.431924 20816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:07.436134 20816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:07.436206 20816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:07.436226 20816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:07.436275 20816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:07.436297 20816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:07.441754 20816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:07.442859 20816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:07.443863 20816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:07.452764 20822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:07.452821 20824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:07.452798 20816 server_base.cc:1034] running on GCE node
W20260924 20:12:07.452764 20821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:07.453588 20816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:07.454798 20816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:07.456054 20816 hybrid_clock.cc:648] HybridClock initialized: now 1790280727456030 us; error 34 us; skew 500 ppm
Sep 24 20:12:07 dist-test-slave-2pm7 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:07.458959 20816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:07.460006 20816 webserver.cc:460] Webserver started at http://127.0.0.1:39793/ using document root <none> and password file <none>
I20260924 20:12:07.460599 20816 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:07.460691 20816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:07.460929 20816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:07.463057 20816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "d051c11cae844273a2ebce83c51fb014"
format_stamp: "Formatted at 2026-09-24 20:12:07 on dist-test-slave-2pm7"
server_key: "7cb3950550ee076a4976f04ca38c58b9"
server_key_iv: "ca76b02144b6ceaa4510403bc1a2cc52"
server_key_version: "encryptionkey@0"
I20260924 20:12:07.463644 20816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "d051c11cae844273a2ebce83c51fb014"
format_stamp: "Formatted at 2026-09-24 20:12:07 on dist-test-slave-2pm7"
server_key: "7cb3950550ee076a4976f04ca38c58b9"
server_key_iv: "ca76b02144b6ceaa4510403bc1a2cc52"
server_key_version: "encryptionkey@0"
I20260924 20:12:07.467595 20816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 20:12:07.470283 20831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:07.471655 20816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:07.471832 20816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "d051c11cae844273a2ebce83c51fb014"
format_stamp: "Formatted at 2026-09-24 20:12:07 on dist-test-slave-2pm7"
server_key: "7cb3950550ee076a4976f04ca38c58b9"
server_key_iv: "ca76b02144b6ceaa4510403bc1a2cc52"
server_key_version: "encryptionkey@0"
I20260924 20:12:07.472020 20816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:07.506953 20816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:07.510675 20816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:07.510900 20816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:07.518633 20816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34047
I20260924 20:12:07.518653 20883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34047 every 8 connection(s)
I20260924 20:12:07.519651 20816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:07.522473 20884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:07.523144 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 20816
I20260924 20:12:07.523298 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:07.523650 462 external_mini_cluster.cc:1428] Setting key 5699bf2f7ac42d40635cda6689a67293
I20260924 20:12:07.529517 20884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014: Bootstrap starting.
I20260924 20:12:07.532259 20884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:07.533229 20884 log.cc:824] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:07.534240 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:07.525186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43727 (local address 127.0.0.1:34047)
0924 20:12:07.525643 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:07.525664 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:07.525710 (+ 46us) server_negotiation.cc:407] Connection header received
0924 20:12:07.526547 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:07.526573 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:07.527023 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:07.527411 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:07.527835 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:07.529336 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:07.530293 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:07.530999 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:07.531174 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:07.532117 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:07.532127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:07.532147 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:07.532702 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:07.532772 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:07.532804 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:07.532940 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:07.533155 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 20:12:07.533302 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":113,"threads_started":1}
I20260924 20:12:07.535563 20884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014: No bootstrap required, opened a new log
I20260924 20:12:07.537382 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:07.539631 20884 raft_consensus.cc:348] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d051c11cae844273a2ebce83c51fb014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260924 20:12:07.539973 20884 raft_consensus.cc:374] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:07.540104 20884 raft_consensus.cc:729] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d051c11cae844273a2ebce83c51fb014, State: Initialized, Role: FOLLOWER
I20260924 20:12:07.540639 20884 consensus_queue.cc:260] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d051c11cae844273a2ebce83c51fb014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260924 20:12:07.540803 20884 raft_consensus.cc:388] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:07.540865 20884 raft_consensus.cc:482] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:07.540958 20884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:07.541954 20884 raft_consensus.cc:504] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d051c11cae844273a2ebce83c51fb014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260924 20:12:07.542325 20884 leader_election.cc:302] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d051c11cae844273a2ebce83c51fb014; no voters:
I20260924 20:12:07.542706 20884 leader_election.cc:288] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:07.542773 20894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:07.542944 20894 raft_consensus.cc:686] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [term 1 LEADER]: Becoming Leader. State: Replica: d051c11cae844273a2ebce83c51fb014, State: Running, Role: LEADER
I20260924 20:12:07.543238 20894 consensus_queue.cc:237] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d051c11cae844273a2ebce83c51fb014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260924 20:12:07.543912 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:07.539444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50403 (local address 127.0.0.1:34047)
0924 20:12:07.539559 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:07.539563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:07.539708 (+ 145us) server_negotiation.cc:407] Connection header received
0924 20:12:07.539795 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:07.539798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:07.539846 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:07.539927 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:07.540305 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:07.541079 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:07.541950 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:07.542293 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:07.542409 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:07.543364 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:07.543367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:07.543370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:07.543386 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:07.543430 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:07.543442 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:07.543538 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:07.543737 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 20:12:07.543774 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:12:07.544071 20884 sys_catalog.cc:564] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:07.545324 20895 sys_catalog.cc:455] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d051c11cae844273a2ebce83c51fb014. Latest consensus state: current_term: 1 leader_uuid: "d051c11cae844273a2ebce83c51fb014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d051c11cae844273a2ebce83c51fb014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260924 20:12:07.545459 20895 sys_catalog.cc:458] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:07.545810 20896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d051c11cae844273a2ebce83c51fb014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d051c11cae844273a2ebce83c51fb014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260924 20:12:07.545866 20903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:07.545925 20896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:07.549263 20903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:07.555173 20903 catalog_manager.cc:1347] Generated new cluster ID: f3d3b40cc3234ef48e54469ec015df28
I20260924 20:12:07.555307 20903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:07.575963 20903 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:07.577291 20903 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:07.587518 20903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d051c11cae844273a2ebce83c51fb014: Generated new TSK 0
I20260924 20:12:07.588308 20903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:07.600785 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 20816
W20260924 20:12:07.607138 20801 connection.cc:535] client connection to 127.0.0.1:34047 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34047): Connection reset by peer (error 104)
2026-09-24T20:12:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:07 dist-test-slave-2pm7 krb5kdc[20915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:07 dist-test-slave-2pm7 krb5kdc[20915](info): set up 2 sockets
Sep 24 20:12:07 dist-test-slave-2pm7 krb5kdc[20915](info): commencing operation
krb5kdc: starting...
W20260924 20:12:08.704417 462 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:08 dist-test-slave-2pm7 krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:08.832206 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:08.944612 20928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:08.944885 20928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:08.944934 20928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:08.949154 20928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:08.949225 20928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:08.949245 20928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:08.949262 20928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:08.949277 20928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:08.954741 20928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:08.955879 20928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:08.956992 20928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:08.965883 20933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:08.966110 20928 server_base.cc:1034] running on GCE node
W20260924 20:12:08.965883 20936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:08.965924 20934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:08.966857 20928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:08.968078 20928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:08.969245 20928 hybrid_clock.cc:648] HybridClock initialized: now 1790280728969228 us; error 30 us; skew 500 ppm
Sep 24 20:12:08 dist-test-slave-2pm7 krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:08.981055 20928 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:08.982113 20928 webserver.cc:460] Webserver started at http://127.0.0.1:43561/ using document root <none> and password file <none>
I20260924 20:12:08.982724 20928 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:08.982791 20928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:08.983023 20928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:08.985260 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "963d34389c484935a0b0398893ffa130"
format_stamp: "Formatted at 2026-09-24 20:12:08 on dist-test-slave-2pm7"
server_key: "1310879b91919bc259702cb6d60218a9"
server_key_iv: "ff7e5cc996884ea4e75666b491bbcdcc"
server_key_version: "encryptionkey@0"
I20260924 20:12:08.985852 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "963d34389c484935a0b0398893ffa130"
format_stamp: "Formatted at 2026-09-24 20:12:08 on dist-test-slave-2pm7"
server_key: "1310879b91919bc259702cb6d60218a9"
server_key_iv: "ff7e5cc996884ea4e75666b491bbcdcc"
server_key_version: "encryptionkey@0"
I20260924 20:12:08.989817 20928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:12:08.992393 20943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:08.993690 20928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:08.993835 20928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "963d34389c484935a0b0398893ffa130"
format_stamp: "Formatted at 2026-09-24 20:12:08 on dist-test-slave-2pm7"
server_key: "1310879b91919bc259702cb6d60218a9"
server_key_iv: "ff7e5cc996884ea4e75666b491bbcdcc"
server_key_version: "encryptionkey@0"
I20260924 20:12:08.993963 20928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:08.998721 20928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:09.002097 20928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:09.002297 20928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:09.010138 20928 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37451
I20260924 20:12:09.010164 20995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37451 every 8 connection(s)
I20260924 20:12:09.011116 20928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:09.013963 20996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:09.018754 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 20928
I20260924 20:12:09.018922 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:09.019244 462 external_mini_cluster.cc:1428] Setting key 393aadb1bbbbb1e8735a069cfc283283
I20260924 20:12:09.019856 20996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130: Bootstrap starting.
I20260924 20:12:09.022351 20996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:09.023221 20996 log.cc:824] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:09.024606 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:09.020706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36129 (local address 127.0.0.1:37451)
0924 20:12:09.021158 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:09.021173 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:09.021208 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:12:09.021854 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:09.021872 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:09.022219 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:09.022559 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:09.023354 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:09.023360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:09.023377 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:09.023388 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:09.023451 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:09.023474 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:09.023592 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:09.023694 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 20:12:09.023851 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260924 20:12:09.025223 20996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130: No bootstrap required, opened a new log
I20260924 20:12:09.027941 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:09.029385 20996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d34389c484935a0b0398893ffa130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260924 20:12:09.029627 20996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:09.029707 20996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963d34389c484935a0b0398893ffa130, State: Initialized, Role: FOLLOWER
I20260924 20:12:09.030189 20996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d34389c484935a0b0398893ffa130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260924 20:12:09.030350 20996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:09.030421 20996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:09.030550 20996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:09.031646 20996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d34389c484935a0b0398893ffa130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260924 20:12:09.031745 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:09.030048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60109 (local address 127.0.0.1:37451)
0924 20:12:09.030157 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:09.030162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:09.030310 (+ 148us) server_negotiation.cc:407] Connection header received
0924 20:12:09.030447 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:09.030450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:09.030528 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:09.030616 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:09.031306 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:09.031309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:09.031312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:09.031330 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:09.031362 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:09.031373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:09.031466 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:09.031542 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 20:12:09.031581 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:12:09.032063 20996 leader_election.cc:302] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 963d34389c484935a0b0398893ffa130; no voters:
I20260924 20:12:09.032408 20996 leader_election.cc:288] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:09.032522 21006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:09.032761 21006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [term 1 LEADER]: Becoming Leader. State: Replica: 963d34389c484935a0b0398893ffa130, State: Running, Role: LEADER
I20260924 20:12:09.033106 21006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d34389c484935a0b0398893ffa130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260924 20:12:09.033636 20996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:09.034865 21008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "963d34389c484935a0b0398893ffa130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d34389c484935a0b0398893ffa130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } } }
I20260924 20:12:09.034870 21007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 963d34389c484935a0b0398893ffa130. Latest consensus state: current_term: 1 leader_uuid: "963d34389c484935a0b0398893ffa130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d34389c484935a0b0398893ffa130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } } }
I20260924 20:12:09.035018 21008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:09.035022 21007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:09.035370 21015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:09.038036 21015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:09.043301 21015 catalog_manager.cc:1347] Generated new cluster ID: 0a4d5117031e49198722e4cdc90c062b
I20260924 20:12:09.043381 21015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:09.052742 21015 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:09.053112 21015 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:09.064976 21015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 963d34389c484935a0b0398893ffa130: Generated new TSK 0
I20260924 20:12:09.065874 21015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:09.074584 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 20928
2026-09-24T20:12:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:09 dist-test-slave-2pm7 krb5kdc[21027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:09 dist-test-slave-2pm7 krb5kdc[21027](info): set up 2 sockets
Sep 24 20:12:09 dist-test-slave-2pm7 krb5kdc[21027](info): commencing operation
krb5kdc: starting...
W20260924 20:12:10.100260 462 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:10 dist-test-slave-2pm7 krb5kdc[21027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:10.222838 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:10.336863 21040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:10.337145 21040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:10.337189 21040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:10.341356 21040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:10.341432 21040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:10.341454 21040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:10.341470 21040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:10.341528 21040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:10.346884 21040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:10.348055 21040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:10.349107 21040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:10.357968 21045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:10.358007 21046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:10.358074 21048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:10.358170 21040 server_base.cc:1034] running on GCE node
I20260924 20:12:10.358876 21040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:10.360148 21040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:10.361308 21040 hybrid_clock.cc:648] HybridClock initialized: now 1790280730361294 us; error 28 us; skew 500 ppm
Sep 24 20:12:10 dist-test-slave-2pm7 krb5kdc[21027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:10.364215 21040 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:10.365116 21040 webserver.cc:460] Webserver started at http://127.0.0.1:38259/ using document root <none> and password file <none>
I20260924 20:12:10.365713 21040 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:10.365804 21040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:10.366058 21040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:10.368255 21040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "da1552fd0620413195f4188fa7f8c4dc"
format_stamp: "Formatted at 2026-09-24 20:12:10 on dist-test-slave-2pm7"
server_key: "0a9a918b1d320fb5a9d2968f1b2ae320"
server_key_iv: "27755f1ac16af411abf39a16563b030c"
server_key_version: "encryptionkey@0"
I20260924 20:12:10.368839 21040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "da1552fd0620413195f4188fa7f8c4dc"
format_stamp: "Formatted at 2026-09-24 20:12:10 on dist-test-slave-2pm7"
server_key: "0a9a918b1d320fb5a9d2968f1b2ae320"
server_key_iv: "27755f1ac16af411abf39a16563b030c"
server_key_version: "encryptionkey@0"
I20260924 20:12:10.372756 21040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:12:10.375423 21055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:10.376922 21040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:12:10.377074 21040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "da1552fd0620413195f4188fa7f8c4dc"
format_stamp: "Formatted at 2026-09-24 20:12:10 on dist-test-slave-2pm7"
server_key: "0a9a918b1d320fb5a9d2968f1b2ae320"
server_key_iv: "27755f1ac16af411abf39a16563b030c"
server_key_version: "encryptionkey@0"
I20260924 20:12:10.377199 21040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:10.386245 21040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:10.389211 21040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:10.389425 21040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:10.396615 21040 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38301
I20260924 20:12:10.396641 21107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38301 every 8 connection(s)
I20260924 20:12:10.397636 21040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:10.399246 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21040
I20260924 20:12:10.399355 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:10.399721 462 external_mini_cluster.cc:1428] Setting key 20b0bba13718259f83f8bca53100c90a
I20260924 20:12:10.400751 21108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:10.406565 21108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc: Bootstrap starting.
Sep 24 20:12:10 dist-test-slave-2pm7 krb5kdc[21027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:12:10.409298 21108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:10.410176 21108 log.cc:824] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:10.412345 21108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc: No bootstrap required, opened a new log
I20260924 20:12:10.414408 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:10.401236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38829 (local address 127.0.0.1:38301)
0924 20:12:10.401707 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:10.401721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:10.401759 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:12:10.402406 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:10.402424 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:10.402778 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:10.403141 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:10.403988 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:10.405098 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:10.406020 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:10.406531 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:10.406702 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:10.409464 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:10.409514 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:10.409529 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:10.409558 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:10.411401 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:10.411891 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:10.411903 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:10.411907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:10.412086 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:10.412378 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:10.412382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:10.412384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:10.412847 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:10.413051 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:10.413356 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 20:12:10.413671 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":157,"threads_started":1}
I20260924 20:12:10.416344 21108 raft_consensus.cc:348] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1552fd0620413195f4188fa7f8c4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38301 } }
I20260924 20:12:10.416616 21108 raft_consensus.cc:374] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:10.416683 21108 raft_consensus.cc:729] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1552fd0620413195f4188fa7f8c4dc, State: Initialized, Role: FOLLOWER
I20260924 20:12:10.417162 21108 consensus_queue.cc:260] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1552fd0620413195f4188fa7f8c4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38301 } }
I20260924 20:12:10.417330 21108 raft_consensus.cc:388] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:10.417445 21108 raft_consensus.cc:482] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:10.417501 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:10.417555 21108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:10.418489 21108 raft_consensus.cc:504] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1552fd0620413195f4188fa7f8c4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38301 } }
I20260924 20:12:10.418864 21108 leader_election.cc:302] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da1552fd0620413195f4188fa7f8c4dc; no voters:
I20260924 20:12:10.419219 21108 leader_election.cc:288] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:10.419297 21117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:10.420073 21117 raft_consensus.cc:686] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [term 1 LEADER]: Becoming Leader. State: Replica: da1552fd0620413195f4188fa7f8c4dc, State: Running, Role: LEADER
I20260924 20:12:10.420374 21117 consensus_queue.cc:237] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1552fd0620413195f4188fa7f8c4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38301 } }
I20260924 20:12:10.420636 21108 sys_catalog.cc:564] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:10.421605 21117 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader da1552fd0620413195f4188fa7f8c4dc. Latest consensus state: current_term: 1 leader_uuid: "da1552fd0620413195f4188fa7f8c4dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1552fd0620413195f4188fa7f8c4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38301 } } }
I20260924 20:12:10.421772 21117 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:10.422096 21118 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da1552fd0620413195f4188fa7f8c4dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1552fd0620413195f4188fa7f8c4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38301 } } }
I20260924 20:12:10.422195 21118 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc [sys.catalog]: This master's current role is: LEADER
W20260924 20:12:10.426635 21133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:10.426748 21133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:10.426877 21127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:10.427875 21127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:10.429236 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:10.419674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37247 (local address 127.0.0.1:38301)
0924 20:12:10.419780 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:10.419784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:10.420043 (+ 259us) server_negotiation.cc:407] Connection header received
0924 20:12:10.420239 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:10.420242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:10.420283 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:10.420370 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:10.421225 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:10.422478 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:10.423612 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:10.423983 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:10.424094 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:10.425681 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:10.425685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:10.425689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:10.425716 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:10.427325 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:10.427830 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:10.427834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:10.427836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:10.427926 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:10.428273 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:10.428277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:10.428279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:10.428522 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:10.428660 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:10.428947 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 20:12:10.429077 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 20:12:10.434633 21127 catalog_manager.cc:1347] Generated new cluster ID: 9da04f3c336d49629f67c29a47f59b4e
I20260924 20:12:10.434746 21127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:10.448001 21127 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:10.449030 21127 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:10.457530 21127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da1552fd0620413195f4188fa7f8c4dc: Generated new TSK 0
I20260924 20:12:10.458299 21127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:10.467985 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21040
2026-09-24T20:12:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:10 dist-test-slave-2pm7 krb5kdc[21139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:10 dist-test-slave-2pm7 krb5kdc[21139](info): set up 2 sockets
Sep 24 20:12:10 dist-test-slave-2pm7 krb5kdc[21139](info): commencing operation
krb5kdc: starting...
W20260924 20:12:11.516894 462 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:11 dist-test-slave-2pm7 krb5kdc[21139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:11.639190 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:11.751266 21152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:11.751546 21152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:11.751587 21152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:11.755779 21152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:11.755837 21152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:11.755856 21152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:11.755873 21152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:11.755888 21152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:11.761360 21152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:11.762440 21152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:11.763558 21152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:11.772184 21158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:11.772198 21157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:11.772184 21160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:11.772620 21152 server_base.cc:1034] running on GCE node
I20260924 20:12:11.773238 21152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:11.774477 21152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:11.775663 21152 hybrid_clock.cc:648] HybridClock initialized: now 1790280731775630 us; error 41 us; skew 500 ppm
Sep 24 20:12:11 dist-test-slave-2pm7 krb5kdc[21139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:11.778887 21152 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:11.779780 21152 webserver.cc:460] Webserver started at http://127.0.0.1:34661/ using document root <none> and password file <none>
I20260924 20:12:11.780388 21152 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:11.780448 21152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:11.780671 21152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:11.782723 21152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "66b886e34c2d41caaa61a2a1cda4c6df"
format_stamp: "Formatted at 2026-09-24 20:12:11 on dist-test-slave-2pm7"
server_key: "67bb83543919c1036e8c561425408493"
server_key_iv: "251452be38092c463bb0bd6d3746dbe9"
server_key_version: "encryptionkey@0"
I20260924 20:12:11.783272 21152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "66b886e34c2d41caaa61a2a1cda4c6df"
format_stamp: "Formatted at 2026-09-24 20:12:11 on dist-test-slave-2pm7"
server_key: "67bb83543919c1036e8c561425408493"
server_key_iv: "251452be38092c463bb0bd6d3746dbe9"
server_key_version: "encryptionkey@0"
I20260924 20:12:11.787778 21152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 20:12:11.790230 21167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:11.791496 21152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:11.791664 21152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "66b886e34c2d41caaa61a2a1cda4c6df"
format_stamp: "Formatted at 2026-09-24 20:12:11 on dist-test-slave-2pm7"
server_key: "67bb83543919c1036e8c561425408493"
server_key_iv: "251452be38092c463bb0bd6d3746dbe9"
server_key_version: "encryptionkey@0"
I20260924 20:12:11.791787 21152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:11.809351 21152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:11.812546 21152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:11.812764 21152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:11.820474 21152 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40501
I20260924 20:12:11.820564 21219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40501 every 8 connection(s)
I20260924 20:12:11.821548 21152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:11.824474 21220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:11.825771 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21152
I20260924 20:12:11.825912 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:11.826251 462 external_mini_cluster.cc:1428] Setting key 4d91a97e1333eb2944a67c3e0f6aaeb9
I20260924 20:12:11.830693 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df: Bootstrap starting.
I20260924 20:12:11.833045 21220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:11.833865 21220 log.cc:824] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:11.835371 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:11.827645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56469 (local address 127.0.0.1:40501)
0924 20:12:11.828170 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:11.828186 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:11.828224 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:12:11.828885 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:11.828903 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:11.829255 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:11.829585 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:11.830011 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:11.831031 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:11.831931 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:11.832471 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:11.832645 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:11.833553 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:11.833558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:11.833572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:11.833954 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:11.834041 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:11.834082 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:11.834190 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:11.834311 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 20:12:11.834549 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":148,"threads_started":1}
I20260924 20:12:11.836287 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df: No bootstrap required, opened a new log
I20260924 20:12:11.839648 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:11.841337 21220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260924 20:12:11.841635 21220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:11.841710 21220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b886e34c2d41caaa61a2a1cda4c6df, State: Initialized, Role: FOLLOWER
I20260924 20:12:11.842260 21220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260924 20:12:11.842427 21220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:11.842511 21220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:11.842613 21220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:11.843777 21220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260924 20:12:11.844192 21220 leader_election.cc:302] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b886e34c2d41caaa61a2a1cda4c6df; no voters:
I20260924 20:12:11.844519 21220 leader_election.cc:288] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:11.844637 21230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:11.844849 21230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [term 1 LEADER]: Becoming Leader. State: Replica: 66b886e34c2d41caaa61a2a1cda4c6df, State: Running, Role: LEADER
I20260924 20:12:11.845155 21230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260924 20:12:11.845878 21220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:11.846751 21231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } } }
I20260924 20:12:11.846905 21231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:11.847265 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:11.841948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33765 (local address 127.0.0.1:40501)
0924 20:12:11.842075 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:11.842082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:11.842398 (+ 316us) server_negotiation.cc:407] Connection header received
0924 20:12:11.842566 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:11.842569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:11.842620 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:11.842691 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:11.842987 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:11.843859 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:11.845051 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:11.845383 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:11.845463 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:11.846460 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:11.846464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:11.846468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:11.846480 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:11.846506 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:11.846519 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:11.846564 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:11.846761 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 20:12:11.846831 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:12:11.847504 21232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b886e34c2d41caaa61a2a1cda4c6df. Latest consensus state: current_term: 1 leader_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b886e34c2d41caaa61a2a1cda4c6df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } } }
I20260924 20:12:11.847601 21232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:11.848441 21239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:11.851094 21239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:11.857234 21239 catalog_manager.cc:1347] Generated new cluster ID: 2694346a46cc42ad80c4beaa6cb6f240
I20260924 20:12:11.857321 21239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:11.869527 21239 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:11.870772 21239 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:11.878360 21239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b886e34c2d41caaa61a2a1cda4c6df: Generated new TSK 0
I20260924 20:12:11.879228 21239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:11.890362 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21152
2026-09-24T20:12:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:11 dist-test-slave-2pm7 krb5kdc[21251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:11 dist-test-slave-2pm7 krb5kdc[21251](info): set up 2 sockets
Sep 24 20:12:11 dist-test-slave-2pm7 krb5kdc[21251](info): commencing operation
krb5kdc: starting...
W20260924 20:12:12.948499 462 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:13 dist-test-slave-2pm7 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:13.075659 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:13.189018 21264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:13.189297 21264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:13.189338 21264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:13.193354 21264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:13.193416 21264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:13.193436 21264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:13.193454 21264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:13.193468 21264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:13.198879 21264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:13.199927 21264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:13.200973 21264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:13.209791 21269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:13.209939 21264 server_base.cc:1034] running on GCE node
W20260924 20:12:13.209811 21270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:13.209811 21272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:13.210631 21264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:13.211900 21264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:13.213156 21264 hybrid_clock.cc:648] HybridClock initialized: now 1790280733213119 us; error 45 us; skew 500 ppm
Sep 24 20:12:13 dist-test-slave-2pm7 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:13.216081 21264 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:13.217141 21264 webserver.cc:460] Webserver started at http://127.0.0.1:33083/ using document root <none> and password file <none>
I20260924 20:12:13.217752 21264 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:13.217849 21264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:13.218084 21264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:13.220281 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "2220cfe4d04f418a8fd6b6078999fef2"
format_stamp: "Formatted at 2026-09-24 20:12:13 on dist-test-slave-2pm7"
server_key: "6480ca22eb91a6133783e4762ab9519f"
server_key_iv: "c035fca7ed335b2d48497590e775b26f"
server_key_version: "encryptionkey@0"
I20260924 20:12:13.220872 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "2220cfe4d04f418a8fd6b6078999fef2"
format_stamp: "Formatted at 2026-09-24 20:12:13 on dist-test-slave-2pm7"
server_key: "6480ca22eb91a6133783e4762ab9519f"
server_key_iv: "c035fca7ed335b2d48497590e775b26f"
server_key_version: "encryptionkey@0"
I20260924 20:12:13.224938 21264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:12:13.227586 21279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:13.229178 21264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:12:13.229329 21264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "2220cfe4d04f418a8fd6b6078999fef2"
format_stamp: "Formatted at 2026-09-24 20:12:13 on dist-test-slave-2pm7"
server_key: "6480ca22eb91a6133783e4762ab9519f"
server_key_iv: "c035fca7ed335b2d48497590e775b26f"
server_key_version: "encryptionkey@0"
I20260924 20:12:13.229547 21264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:13.234246 21264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:13.237313 21264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:13.237481 21264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:13.244894 21264 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42083
I20260924 20:12:13.244920 21331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42083 every 8 connection(s)
I20260924 20:12:13.245836 21264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:13.248605 21332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:13.252198 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21264
I20260924 20:12:13.252334 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:13.252647 462 external_mini_cluster.cc:1428] Setting key 4eaae008c1bb8c391da9ce5c00937bb5
I20260924 20:12:13.254616 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2: Bootstrap starting.
I20260924 20:12:13.257197 21332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:13.257956 21332 log.cc:824] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:13.258085 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:13.254085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60311 (local address 127.0.0.1:42083)
0924 20:12:13.254518 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:13.254537 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:13.254575 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:12:13.255277 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:13.255297 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:13.255594 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:13.255877 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:13.256733 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:13.256738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:13.256751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:13.256760 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:13.256811 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:13.256830 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:13.256921 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:13.257073 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 20:12:13.257250 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":129,"threads_started":1}
I20260924 20:12:13.260244 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2: No bootstrap required, opened a new log
I20260924 20:12:13.261137 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:13.264684 21332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2220cfe4d04f418a8fd6b6078999fef2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260924 20:12:13.264946 21332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:13.265031 21332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2220cfe4d04f418a8fd6b6078999fef2, State: Initialized, Role: FOLLOWER
I20260924 20:12:13.265183 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:13.263342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40927 (local address 127.0.0.1:42083)
0924 20:12:13.263490 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:13.263494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:13.263709 (+ 215us) server_negotiation.cc:407] Connection header received
0924 20:12:13.263862 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:13.263865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:13.263903 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:13.264015 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:13.264753 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:13.264757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:13.264759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:13.264771 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:13.264798 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:13.264819 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:13.264911 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:13.265023 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 20:12:13.265065 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:12:13.265528 21332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2220cfe4d04f418a8fd6b6078999fef2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260924 20:12:13.265692 21332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:13.265767 21332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:13.265868 21332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:13.266865 21332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2220cfe4d04f418a8fd6b6078999fef2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260924 20:12:13.267159 21332 leader_election.cc:302] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2220cfe4d04f418a8fd6b6078999fef2; no voters:
I20260924 20:12:13.267391 21332 leader_election.cc:288] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:13.267518 21342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:13.267763 21342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [term 1 LEADER]: Becoming Leader. State: Replica: 2220cfe4d04f418a8fd6b6078999fef2, State: Running, Role: LEADER
I20260924 20:12:13.268131 21342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2220cfe4d04f418a8fd6b6078999fef2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260924 20:12:13.268596 21332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:13.269840 21344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2220cfe4d04f418a8fd6b6078999fef2. Latest consensus state: current_term: 1 leader_uuid: "2220cfe4d04f418a8fd6b6078999fef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2220cfe4d04f418a8fd6b6078999fef2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } } }
I20260924 20:12:13.269996 21344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:13.269840 21343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2220cfe4d04f418a8fd6b6078999fef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2220cfe4d04f418a8fd6b6078999fef2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } } }
I20260924 20:12:13.270269 21343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:13.270431 21352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:13.273697 21352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:13.279201 21352 catalog_manager.cc:1347] Generated new cluster ID: e75de62e2da64bf999f482269ed8d027
I20260924 20:12:13.279275 21352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:13.286506 21352 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:13.287030 21352 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:13.298431 21352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2220cfe4d04f418a8fd6b6078999fef2: Generated new TSK 0
I20260924 20:12:13.299175 21352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:13.306764 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21264
2026-09-24T20:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:13 dist-test-slave-2pm7 krb5kdc[21363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:13 dist-test-slave-2pm7 krb5kdc[21363](info): set up 2 sockets
Sep 24 20:12:13 dist-test-slave-2pm7 krb5kdc[21363](info): commencing operation
krb5kdc: starting...
W20260924 20:12:14.352451 462 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:14 dist-test-slave-2pm7 krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:14.477120 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:14.591529 21376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:14.591813 21376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:14.591856 21376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:14.596670 21376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:14.596738 21376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:14.596760 21376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:14.596776 21376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:14.596791 21376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:14.602210 21376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:14.603284 21376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:14.604344 21376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:14.612954 21384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:14.612977 21381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:14.612982 21382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:14.613245 21376 server_base.cc:1034] running on GCE node
I20260924 20:12:14.613867 21376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:14.615049 21376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:14.616238 21376 hybrid_clock.cc:648] HybridClock initialized: now 1790280734616210 us; error 49 us; skew 500 ppm
Sep 24 20:12:14 dist-test-slave-2pm7 krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:14.625422 21376 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:14.626466 21376 webserver.cc:460] Webserver started at http://127.0.0.1:34837/ using document root <none> and password file <none>
I20260924 20:12:14.627079 21376 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:14.627174 21376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:14.627414 21376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:14.629568 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "14b711bd8a29469f82100de5c25edcbe"
format_stamp: "Formatted at 2026-09-24 20:12:14 on dist-test-slave-2pm7"
server_key: "c76edbc4cb1a6b7b00a488f30e9e3373"
server_key_iv: "398db8ff5d88e2fb01b0427571e14a7f"
server_key_version: "encryptionkey@0"
I20260924 20:12:14.630146 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "14b711bd8a29469f82100de5c25edcbe"
format_stamp: "Formatted at 2026-09-24 20:12:14 on dist-test-slave-2pm7"
server_key: "c76edbc4cb1a6b7b00a488f30e9e3373"
server_key_iv: "398db8ff5d88e2fb01b0427571e14a7f"
server_key_version: "encryptionkey@0"
I20260924 20:12:14.633896 21376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 20:12:14.636432 21391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:14.637679 21376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 20:12:14.637823 21376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "14b711bd8a29469f82100de5c25edcbe"
format_stamp: "Formatted at 2026-09-24 20:12:14 on dist-test-slave-2pm7"
server_key: "c76edbc4cb1a6b7b00a488f30e9e3373"
server_key_iv: "398db8ff5d88e2fb01b0427571e14a7f"
server_key_version: "encryptionkey@0"
I20260924 20:12:14.637940 21376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:14.660523 21376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:14.663492 21376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:14.663698 21376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:14.670929 21376 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42075
I20260924 20:12:14.670956 21443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42075 every 8 connection(s)
I20260924 20:12:14.671888 21376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:14.674414 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21376
I20260924 20:12:14.674567 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:14.674911 462 external_mini_cluster.cc:1428] Setting key ed44f1eee13041512a8ea2d924b41959
I20260924 20:12:14.674963 21444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:14.681391 21444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe: Bootstrap starting.
I20260924 20:12:14.683902 21444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe: Neither blocks nor log segments found. Creating new log.
Sep 24 20:12:14 dist-test-slave-2pm7 krb5kdc[21363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:12:14.684805 21444 log.cc:824] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:14.686940 21444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe: No bootstrap required, opened a new log
I20260924 20:12:14.689404 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:14.676347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:56719 (local address 127.0.0.1:42075)
0924 20:12:14.676814 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:14.676830 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:14.676866 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:12:14.677723 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:14.677743 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:14.678085 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:14.678418 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:14.679232 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:14.680291 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:14.681152 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:14.681681 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:14.681869 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:14.684638 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:14.684644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:14.684658 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:14.684684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:14.686511 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:14.687102 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:14.687107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:14.687111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:14.687217 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:14.687568 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:14.687570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:14.687571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:14.687931 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:14.688150 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:14.688456 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 20:12:14.688690 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":155,"threads_started":1}
I20260924 20:12:14.691046 21444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b711bd8a29469f82100de5c25edcbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260924 20:12:14.691331 21444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:14.691421 21444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b711bd8a29469f82100de5c25edcbe, State: Initialized, Role: FOLLOWER
I20260924 20:12:14.691892 21444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b711bd8a29469f82100de5c25edcbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260924 20:12:14.692108 21444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:14.692184 21444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:14.692301 21444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:14.692629 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:14.693403 21444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b711bd8a29469f82100de5c25edcbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260924 20:12:14.693760 21444 leader_election.cc:302] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14b711bd8a29469f82100de5c25edcbe; no voters:
I20260924 20:12:14.694116 21444 leader_election.cc:288] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:14.694216 21452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:14.694521 21452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [term 1 LEADER]: Becoming Leader. State: Replica: 14b711bd8a29469f82100de5c25edcbe, State: Running, Role: LEADER
I20260924 20:12:14.694856 21452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b711bd8a29469f82100de5c25edcbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } }
I20260924 20:12:14.695524 21444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:14.696734 21455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14b711bd8a29469f82100de5c25edcbe. Latest consensus state: current_term: 1 leader_uuid: "14b711bd8a29469f82100de5c25edcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b711bd8a29469f82100de5c25edcbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } } }
I20260924 20:12:14.696734 21454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14b711bd8a29469f82100de5c25edcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b711bd8a29469f82100de5c25edcbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42075 } } }
I20260924 20:12:14.696897 21454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:14.696897 21455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:14.697278 21463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:14.700523 21463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:14.704903 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:14.694884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:46791 (local address 127.0.0.1:42075)
0924 20:12:14.695012 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:14.695016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:14.695408 (+ 392us) server_negotiation.cc:407] Connection header received
0924 20:12:14.695503 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:14.695506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:14.695563 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:14.695641 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:14.696643 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:14.697501 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:14.698801 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:14.699146 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:14.699219 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:14.701091 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:14.701098 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:14.701102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:14.701135 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:14.702781 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:14.703337 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:14.703341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:14.703343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:14.703432 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:14.703812 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:14.703815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:14.703817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:14.704075 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:14.704213 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:14.704637 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 20:12:14.704745 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:12:14.706698 21463 catalog_manager.cc:1347] Generated new cluster ID: 7fb6e5ffe8f44771a22eb98c3da8fbf3
I20260924 20:12:14.706761 21463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:14.730126 21463 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:14.731148 21463 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:14.736943 21463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14b711bd8a29469f82100de5c25edcbe: Generated new TSK 0
I20260924 20:12:14.737775 21463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:14.749661 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21376
W20260924 20:12:14.755497 21361 connection.cc:535] client connection to 127.0.0.1:42075 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42075): Connection reset by peer (error 104)
2026-09-24T20:12:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:14 dist-test-slave-2pm7 krb5kdc[21475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:14 dist-test-slave-2pm7 krb5kdc[21475](info): set up 2 sockets
Sep 24 20:12:14 dist-test-slave-2pm7 krb5kdc[21475](info): commencing operation
krb5kdc: starting...
W20260924 20:12:15.788447 462 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:15 dist-test-slave-2pm7 krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:15.915561 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:16.030979 21488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:16.031250 21488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:16.031293 21488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:16.035339 21488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:16.035403 21488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:16.035422 21488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:16.035439 21488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:16.035454 21488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:16.040860 21488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:16.041983 21488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:16.043016 21488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:16.051867 21496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:16.051867 21493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:16.052173 21488 server_base.cc:1034] running on GCE node
W20260924 20:12:16.051873 21494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:16.052875 21488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:16.054111 21488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:16.055279 21488 hybrid_clock.cc:648] HybridClock initialized: now 1790280736055261 us; error 27 us; skew 500 ppm
Sep 24 20:12:16 dist-test-slave-2pm7 krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:16.058281 21488 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:16.059348 21488 webserver.cc:460] Webserver started at http://127.0.0.1:41585/ using document root <none> and password file <none>
I20260924 20:12:16.060034 21488 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:16.060106 21488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:16.060364 21488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:16.062588 21488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "e411e11114df469ca32c33479cea6d45"
format_stamp: "Formatted at 2026-09-24 20:12:16 on dist-test-slave-2pm7"
server_key: "562a914b636940cc2f5ceed5d47783dd"
server_key_iv: "7b13e3204b1617602baf9e95176f756e"
server_key_version: "encryptionkey@0"
I20260924 20:12:16.063197 21488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "e411e11114df469ca32c33479cea6d45"
format_stamp: "Formatted at 2026-09-24 20:12:16 on dist-test-slave-2pm7"
server_key: "562a914b636940cc2f5ceed5d47783dd"
server_key_iv: "7b13e3204b1617602baf9e95176f756e"
server_key_version: "encryptionkey@0"
I20260924 20:12:16.067242 21488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:12:16.069836 21503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:16.071122 21488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:16.071273 21488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "e411e11114df469ca32c33479cea6d45"
format_stamp: "Formatted at 2026-09-24 20:12:16 on dist-test-slave-2pm7"
server_key: "562a914b636940cc2f5ceed5d47783dd"
server_key_iv: "7b13e3204b1617602baf9e95176f756e"
server_key_version: "encryptionkey@0"
I20260924 20:12:16.071399 21488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:16.095942 21488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:16.099488 21488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:16.099713 21488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:16.106961 21488 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38053
I20260924 20:12:16.106973 21555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38053 every 8 connection(s)
I20260924 20:12:16.107975 21488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:16.110776 21556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:16.112243 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21488
I20260924 20:12:16.112376 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:16.112716 462 external_mini_cluster.cc:1428] Setting key 7c00bb6149436ae60576c4fffe5da9f7
I20260924 20:12:16.117224 21556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45: Bootstrap starting.
I20260924 20:12:16.119993 21556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:16.120802 21556 log.cc:824] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:16.121995 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:16.114101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:40293 (local address 127.0.0.1:38053)
0924 20:12:16.114545 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:16.114562 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:16.114599 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:12:16.115241 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:16.115258 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:16.115599 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:16.115886 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:16.116299 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:16.117248 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:16.118456 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:16.118931 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:16.119049 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:16.119926 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:16.119931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:16.119967 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:16.120500 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:16.120553 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:16.120576 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:16.120692 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:16.120930 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 20:12:16.121068 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
I20260924 20:12:16.122887 21556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45: No bootstrap required, opened a new log
I20260924 20:12:16.124980 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:16.126892 21556 raft_consensus.cc:348] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e411e11114df469ca32c33479cea6d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 20:12:16.127190 21556 raft_consensus.cc:374] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:16.127259 21556 raft_consensus.cc:729] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e411e11114df469ca32c33479cea6d45, State: Initialized, Role: FOLLOWER
I20260924 20:12:16.127760 21556 consensus_queue.cc:260] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e411e11114df469ca32c33479cea6d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 20:12:16.127915 21556 raft_consensus.cc:388] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:16.128019 21556 raft_consensus.cc:482] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:16.128119 21556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:16.129067 21556 raft_consensus.cc:504] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e411e11114df469ca32c33479cea6d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 20:12:16.129415 21556 leader_election.cc:302] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e411e11114df469ca32c33479cea6d45; no voters:
I20260924 20:12:16.129753 21556 leader_election.cc:288] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:16.129848 21566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:16.130038 21566 raft_consensus.cc:686] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [term 1 LEADER]: Becoming Leader. State: Replica: e411e11114df469ca32c33479cea6d45, State: Running, Role: LEADER
I20260924 20:12:16.130333 21566 consensus_queue.cc:237] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e411e11114df469ca32c33479cea6d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 20:12:16.131105 21556 sys_catalog.cc:564] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:16.131525 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:16.127019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:51287 (local address 127.0.0.1:38053)
0924 20:12:16.127138 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:16.127142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:16.127303 (+ 161us) server_negotiation.cc:407] Connection header received
0924 20:12:16.127444 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:16.127447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:16.127486 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:16.127579 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:16.127865 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:16.128541 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:16.129472 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:16.129765 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:16.129889 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:16.130995 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:16.130999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:16.131002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:16.131015 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:16.131041 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:16.131053 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:16.131130 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:16.131331 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 20:12:16.131368 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:12:16.131505 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e411e11114df469ca32c33479cea6d45. Latest consensus state: current_term: 1 leader_uuid: "e411e11114df469ca32c33479cea6d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e411e11114df469ca32c33479cea6d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } } }
I20260924 20:12:16.131605 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:16.132323 21568 sys_catalog.cc:455] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e411e11114df469ca32c33479cea6d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e411e11114df469ca32c33479cea6d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } } }
I20260924 20:12:16.132428 21568 sys_catalog.cc:458] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:16.132714 21570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:16.136065 21570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:16.142112 21570 catalog_manager.cc:1347] Generated new cluster ID: 07b7a1282d324f6f933a63fc5c70f190
I20260924 20:12:16.142206 21570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:16.152076 21570 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:16.153071 21570 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:16.160075 21570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e411e11114df469ca32c33479cea6d45: Generated new TSK 0
I20260924 20:12:16.160799 21570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:16.171769 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21488
W20260924 20:12:16.178037 21473 connection.cc:535] client connection to 127.0.0.1:38053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38053): Connection reset by peer (error 104)
2026-09-24T20:12:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:16 dist-test-slave-2pm7 krb5kdc[21587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:16 dist-test-slave-2pm7 krb5kdc[21587](info): set up 2 sockets
Sep 24 20:12:16 dist-test-slave-2pm7 krb5kdc[21587](info): commencing operation
krb5kdc: starting...
W20260924 20:12:17.216372 462 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:17 dist-test-slave-2pm7 krb5kdc[21587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:17.339480 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:17.452415 21600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:17.452746 21600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:17.452823 21600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:17.457001 21600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:17.457098 21600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:17.457136 21600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:17.457198 21600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:17.457239 21600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:17.462697 21600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:17.463876 21600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:17.464987 21600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:17.474117 21608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:17.474191 21606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:17.474115 21605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:17.474327 21600 server_base.cc:1034] running on GCE node
I20260924 20:12:17.475018 21600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:17.476282 21600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:17.477527 21600 hybrid_clock.cc:648] HybridClock initialized: now 1790280737477496 us; error 44 us; skew 500 ppm
Sep 24 20:12:17 dist-test-slave-2pm7 krb5kdc[21587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:17.480295 21600 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:17.481334 21600 webserver.cc:460] Webserver started at http://127.0.0.1:43369/ using document root <none> and password file <none>
I20260924 20:12:17.482008 21600 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:17.482103 21600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:17.482479 21600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:17.485252 21600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "9c02fc7d33404a0fb795d8fcbe6710ee"
format_stamp: "Formatted at 2026-09-24 20:12:17 on dist-test-slave-2pm7"
server_key: "3d9b4e916220a7aa75487780180135a9"
server_key_iv: "3e09181bd66a56fd556f562316a4fb99"
server_key_version: "encryptionkey@0"
I20260924 20:12:17.485993 21600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "9c02fc7d33404a0fb795d8fcbe6710ee"
format_stamp: "Formatted at 2026-09-24 20:12:17 on dist-test-slave-2pm7"
server_key: "3d9b4e916220a7aa75487780180135a9"
server_key_iv: "3e09181bd66a56fd556f562316a4fb99"
server_key_version: "encryptionkey@0"
I20260924 20:12:17.491219 21600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 20:12:17.495784 21615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:17.497267 21600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:12:17.497416 21600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "9c02fc7d33404a0fb795d8fcbe6710ee"
format_stamp: "Formatted at 2026-09-24 20:12:17 on dist-test-slave-2pm7"
server_key: "3d9b4e916220a7aa75487780180135a9"
server_key_iv: "3e09181bd66a56fd556f562316a4fb99"
server_key_version: "encryptionkey@0"
I20260924 20:12:17.497555 21600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:17.502290 21600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:17.505352 21600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:17.505563 21600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:17.512589 21600 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45961
I20260924 20:12:17.512663 21667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45961 every 8 connection(s)
I20260924 20:12:17.513628 21600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:17.516834 21668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:17.518679 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21600
I20260924 20:12:17.518831 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:17.519152 462 external_mini_cluster.cc:1428] Setting key 17b164bb480a8d805f625daa322b1f83
I20260924 20:12:17.524052 21668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee: Bootstrap starting.
I20260924 20:12:17.524832 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:17.520766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:49129 (local address 127.0.0.1:45961)
0924 20:12:17.521283 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:17.521298 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:17.521335 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:12:17.521987 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:17.522005 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:17.522307 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:17.522654 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:17.523524 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:17.523528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:17.523541 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:17.523552 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:17.523608 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:17.523628 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:17.523784 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:17.523856 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 20:12:17.524070 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":146,"threads_started":1}
I20260924 20:12:17.526826 21668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:17.527804 21668 log.cc:824] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:17.528050 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:17.530453 21668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee: No bootstrap required, opened a new log
I20260924 20:12:17.532549 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:17.530422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:54179 (local address 127.0.0.1:45961)
0924 20:12:17.530555 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:17.530560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:17.530982 (+ 422us) server_negotiation.cc:407] Connection header received
0924 20:12:17.531157 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:17.531160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:17.531202 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:17.531285 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:17.532147 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:17.532151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:17.532153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:17.532165 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:17.532198 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:17.532209 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:17.532299 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:17.532372 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 20:12:17.532415 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:12:17.534865 21668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45961 } }
I20260924 20:12:17.535095 21668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:17.535130 21668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c02fc7d33404a0fb795d8fcbe6710ee, State: Initialized, Role: FOLLOWER
I20260924 20:12:17.535584 21668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45961 } }
I20260924 20:12:17.535712 21668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:17.535753 21668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:17.535827 21668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:17.536778 21668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45961 } }
I20260924 20:12:17.537077 21668 leader_election.cc:302] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c02fc7d33404a0fb795d8fcbe6710ee; no voters:
I20260924 20:12:17.537338 21668 leader_election.cc:288] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:17.537469 21678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:17.537725 21678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [term 1 LEADER]: Becoming Leader. State: Replica: 9c02fc7d33404a0fb795d8fcbe6710ee, State: Running, Role: LEADER
I20260924 20:12:17.538333 21678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45961 } }
I20260924 20:12:17.538777 21668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:17.540228 21680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c02fc7d33404a0fb795d8fcbe6710ee. Latest consensus state: current_term: 1 leader_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45961 } } }
I20260924 20:12:17.540225 21679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c02fc7d33404a0fb795d8fcbe6710ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45961 } } }
I20260924 20:12:17.540382 21680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:17.540382 21679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:17.540868 21691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:17.544446 21691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:17.549773 21691 catalog_manager.cc:1347] Generated new cluster ID: 38d44f42278240ed88b32cc1548edd26
I20260924 20:12:17.549846 21691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:17.566622 21691 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:17.566972 21691 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:17.575551 21691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c02fc7d33404a0fb795d8fcbe6710ee: Generated new TSK 0
I20260924 20:12:17.576432 21691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:17.581290 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21600
2026-09-24T20:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:17 dist-test-slave-2pm7 krb5kdc[21699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:17 dist-test-slave-2pm7 krb5kdc[21699](info): set up 2 sockets
Sep 24 20:12:17 dist-test-slave-2pm7 krb5kdc[21699](info): commencing operation
krb5kdc: starting...
W20260924 20:12:18.612422 462 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:18 dist-test-slave-2pm7 krb5kdc[21699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:18.736666 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:18.849634 21712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:18.849958 21712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:18.850045 21712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:18.854148 21712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:18.854244 21712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:18.854282 21712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:18.854342 21712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:18.854383 21712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:18.859848 21712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:18.861042 21712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:18.862079 21712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:18.871114 21717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:18.871227 21712 server_base.cc:1034] running on GCE node
W20260924 20:12:18.871117 21720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:18.871469 21718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:18.872112 21712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:18.873193 21712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:18.874356 21712 hybrid_clock.cc:648] HybridClock initialized: now 1790280738874321 us; error 48 us; skew 500 ppm
Sep 24 20:12:18 dist-test-slave-2pm7 krb5kdc[21699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:18.877122 21712 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:18.878029 21712 webserver.cc:460] Webserver started at http://127.0.0.1:36033/ using document root <none> and password file <none>
I20260924 20:12:18.878634 21712 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:18.878728 21712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:18.878970 21712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:18.881155 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a"
format_stamp: "Formatted at 2026-09-24 20:12:18 on dist-test-slave-2pm7"
server_key: "bfed7f0e6adf1908c8d5673da8b0e13c"
server_key_iv: "1e67c98728c78d865bd90c8cbd8d8316"
server_key_version: "encryptionkey@0"
I20260924 20:12:18.881752 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a"
format_stamp: "Formatted at 2026-09-24 20:12:18 on dist-test-slave-2pm7"
server_key: "bfed7f0e6adf1908c8d5673da8b0e13c"
server_key_iv: "1e67c98728c78d865bd90c8cbd8d8316"
server_key_version: "encryptionkey@0"
I20260924 20:12:18.885732 21712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:12:18.888267 21727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:18.889521 21712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:12:18.889679 21712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a"
format_stamp: "Formatted at 2026-09-24 20:12:18 on dist-test-slave-2pm7"
server_key: "bfed7f0e6adf1908c8d5673da8b0e13c"
server_key_iv: "1e67c98728c78d865bd90c8cbd8d8316"
server_key_version: "encryptionkey@0"
I20260924 20:12:18.889802 21712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:18.919829 21712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:18.923337 21712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:18.923573 21712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:18.931172 21712 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46109
I20260924 20:12:18.931181 21779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46109 every 8 connection(s)
I20260924 20:12:18.932237 21712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:18.933614 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21712
I20260924 20:12:18.933761 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:18.934120 462 external_mini_cluster.cc:1428] Setting key 95c7552440f53322e2ff4d17829acb16
I20260924 20:12:18.935973 21780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:18.941763 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a: Bootstrap starting.
Sep 24 20:12:18 dist-test-slave-2pm7 krb5kdc[21699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:12:18.944489 21780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:18.945389 21780 log.cc:824] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:18.947953 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a: No bootstrap required, opened a new log
I20260924 20:12:18.949100 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:18.935611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:46197 (local address 127.0.0.1:46109)
0924 20:12:18.936168 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:18.936186 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:18.936223 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:12:18.936907 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:18.936924 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:18.937285 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:18.937626 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:18.938450 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:18.939453 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:18.940404 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:18.940881 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:18.941079 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:18.943888 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:18.943893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:18.943907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:18.943935 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:18.945795 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:18.946559 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:18.946565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:18.946569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:18.946678 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:18.947090 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:18.947092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:18.947093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:18.947476 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:18.947639 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:18.948096 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 20:12:18.948363 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":137,"threads_started":1}
I20260924 20:12:18.951134 21780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46109 } }
I20260924 20:12:18.951400 21780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:18.951462 21780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b8124f8ffa4e7bb8337a36a0c3d91a, State: Initialized, Role: FOLLOWER
I20260924 20:12:18.951951 21780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46109 } }
I20260924 20:12:18.952147 21780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:18.952214 21780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:18.952322 21780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:18.952538 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:18.953579 21780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46109 } }
I20260924 20:12:18.953974 21780 leader_election.cc:302] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82b8124f8ffa4e7bb8337a36a0c3d91a; no voters:
I20260924 20:12:18.954334 21780 leader_election.cc:288] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:18.954463 21789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:18.954703 21789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [term 1 LEADER]: Becoming Leader. State: Replica: 82b8124f8ffa4e7bb8337a36a0c3d91a, State: Running, Role: LEADER
I20260924 20:12:18.955055 21789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46109 } }
I20260924 20:12:18.955796 21780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:18.957063 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46109 } } }
I20260924 20:12:18.957252 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:18.957537 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b8124f8ffa4e7bb8337a36a0c3d91a. Latest consensus state: current_term: 1 leader_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b8124f8ffa4e7bb8337a36a0c3d91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46109 } } }
I20260924 20:12:18.957679 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:18.958009 21800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:18.961589 21800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:18.964709 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:18.955213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:46665 (local address 127.0.0.1:46109)
0924 20:12:18.955316 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:18.955320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:18.955335 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:12:18.955375 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:18.955377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:18.955430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:18.955496 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:18.956413 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:18.957277 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:18.958685 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:18.959034 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:18.959146 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:18.960944 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:18.960947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:18.960950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:18.960977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:18.962664 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:18.963248 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:18.963253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:18.963255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:18.963342 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:18.963701 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:18.963704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:18.963706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:18.963924 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:18.964103 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:18.964449 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 20:12:18.964550 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 20:12:18.968381 21800 catalog_manager.cc:1347] Generated new cluster ID: 6f3635a745cc4a49bdfc7451f03f480f
I20260924 20:12:18.968461 21800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:18.977586 21800 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:18.978521 21800 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:18.990470 21800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82b8124f8ffa4e7bb8337a36a0c3d91a: Generated new TSK 0
I20260924 20:12:18.991232 21800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:19.002645 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21712
W20260924 20:12:19.008347 21697 connection.cc:535] client connection to 127.0.0.1:46109 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46109): Connection reset by peer (error 104)
2026-09-24T20:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:19 dist-test-slave-2pm7 krb5kdc[21811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:19 dist-test-slave-2pm7 krb5kdc[21811](info): set up 2 sockets
Sep 24 20:12:19 dist-test-slave-2pm7 krb5kdc[21811](info): commencing operation
krb5kdc: starting...
W20260924 20:12:20.052304 462 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:20 dist-test-slave-2pm7 krb5kdc[21811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:20.176573 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:20.291385 21824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:20.291666 21824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:20.291747 21824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:20.295930 21824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:20.296022 21824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:20.296041 21824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:20.296109 21824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:20.296139 21824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:20.301612 21824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:20.302881 21824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:20.303975 21824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:20.312808 21829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:20.312808 21830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:20.312863 21824 server_base.cc:1034] running on GCE node
W20260924 20:12:20.313090 21832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:20.313812 21824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:20.315268 21824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:20.316493 21824 hybrid_clock.cc:648] HybridClock initialized: now 1790280740316477 us; error 41 us; skew 500 ppm
Sep 24 20:12:20 dist-test-slave-2pm7 krb5kdc[21811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:20.325418 21824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:20.326498 21824 webserver.cc:460] Webserver started at http://127.0.0.1:38335/ using document root <none> and password file <none>
I20260924 20:12:20.327127 21824 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:20.327227 21824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:20.327487 21824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:20.329789 21824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "93152ae2f52f475ba835b57b49b1b614"
format_stamp: "Formatted at 2026-09-24 20:12:20 on dist-test-slave-2pm7"
server_key: "bf0bf28a21c045bb782f560bc39d2f9e"
server_key_iv: "2da1ba8e08432cb3fffe8ee1d6d95f17"
server_key_version: "encryptionkey@0"
I20260924 20:12:20.330379 21824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "93152ae2f52f475ba835b57b49b1b614"
format_stamp: "Formatted at 2026-09-24 20:12:20 on dist-test-slave-2pm7"
server_key: "bf0bf28a21c045bb782f560bc39d2f9e"
server_key_iv: "2da1ba8e08432cb3fffe8ee1d6d95f17"
server_key_version: "encryptionkey@0"
I20260924 20:12:20.334384 21824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 20:12:20.337095 21839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:20.338440 21824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:12:20.338656 21824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "93152ae2f52f475ba835b57b49b1b614"
format_stamp: "Formatted at 2026-09-24 20:12:20 on dist-test-slave-2pm7"
server_key: "bf0bf28a21c045bb782f560bc39d2f9e"
server_key_iv: "2da1ba8e08432cb3fffe8ee1d6d95f17"
server_key_version: "encryptionkey@0"
I20260924 20:12:20.338788 21824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:20.350612 21824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:20.354094 21824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:20.354321 21824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:20.361688 21824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42129
I20260924 20:12:20.361707 21891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42129 every 8 connection(s)
I20260924 20:12:20.362716 21824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:20.363636 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21824
I20260924 20:12:20.363765 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:20.364143 462 external_mini_cluster.cc:1428] Setting key 9521d8a00bea6f9152057c21e9b705b4
I20260924 20:12:20.366267 21892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:20.372884 21892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614: Bootstrap starting.
I20260924 20:12:20.373847 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:20.365649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:40827 (local address 127.0.0.1:42129)
0924 20:12:20.366149 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:20.366164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:20.366407 (+ 243us) server_negotiation.cc:407] Connection header received
0924 20:12:20.367203 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:20.367226 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:20.367600 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:20.367887 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:20.368294 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:20.369317 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:20.370522 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:20.371022 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:20.371151 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:20.372109 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:20.372114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:20.372127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:20.372511 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:20.372573 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:20.372596 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:20.372698 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:20.372912 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 20:12:20.373072 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":146,"threads_started":1}
I20260924 20:12:20.375613 21892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:20.376542 21892 log.cc:824] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:20.377095 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:20.378930 21892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614: No bootstrap required, opened a new log
I20260924 20:12:20.383721 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:20.379389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:37839 (local address 127.0.0.1:42129)
0924 20:12:20.379519 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:20.379523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:20.379734 (+ 211us) server_negotiation.cc:407] Connection header received
0924 20:12:20.379857 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:20.379860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:20.379899 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:20.380032 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:20.380300 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:20.380804 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:20.382009 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:20.382237 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:20.382311 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:20.383212 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:20.383215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:20.383217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:20.383230 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:20.383254 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:20.383265 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:20.383344 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:20.383497 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 20:12:20.383530 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:12:20.383903 21892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93152ae2f52f475ba835b57b49b1b614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260924 20:12:20.384243 21892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:20.384315 21892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93152ae2f52f475ba835b57b49b1b614, State: Initialized, Role: FOLLOWER
I20260924 20:12:20.384802 21892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93152ae2f52f475ba835b57b49b1b614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260924 20:12:20.384958 21892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:20.385026 21892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:20.385134 21892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:20.386099 21892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93152ae2f52f475ba835b57b49b1b614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260924 20:12:20.386468 21892 leader_election.cc:302] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93152ae2f52f475ba835b57b49b1b614; no voters:
I20260924 20:12:20.386801 21892 leader_election.cc:288] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:20.386955 21902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:20.387197 21902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [term 1 LEADER]: Becoming Leader. State: Replica: 93152ae2f52f475ba835b57b49b1b614, State: Running, Role: LEADER
I20260924 20:12:20.387634 21902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93152ae2f52f475ba835b57b49b1b614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260924 20:12:20.388204 21892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:20.389647 21904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93152ae2f52f475ba835b57b49b1b614. Latest consensus state: current_term: 1 leader_uuid: "93152ae2f52f475ba835b57b49b1b614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93152ae2f52f475ba835b57b49b1b614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } } }
I20260924 20:12:20.389784 21904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:20.390106 21903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93152ae2f52f475ba835b57b49b1b614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93152ae2f52f475ba835b57b49b1b614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } } }
I20260924 20:12:20.390182 21912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:20.390228 21903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:20.393625 21912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:20.400226 21912 catalog_manager.cc:1347] Generated new cluster ID: 3be8887e188842dd98ec53787a9a28f1
I20260924 20:12:20.400300 21912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:20.411017 21912 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:20.412060 21912 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:20.423429 21912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93152ae2f52f475ba835b57b49b1b614: Generated new TSK 0
I20260924 20:12:20.424279 21912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:20.438968 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21824
W20260924 20:12:20.445243 21809 connection.cc:535] client connection to 127.0.0.1:42129 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42129): Connection reset by peer (error 104)
2026-09-24T20:12:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:20 dist-test-slave-2pm7 krb5kdc[21923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:20 dist-test-slave-2pm7 krb5kdc[21923](info): set up 2 sockets
Sep 24 20:12:20 dist-test-slave-2pm7 krb5kdc[21923](info): commencing operation
krb5kdc: starting...
W20260924 20:12:21.508396 462 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:21 dist-test-slave-2pm7 krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:12:21.634564 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:21.747088 21936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:21.747359 21936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:21.747400 21936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:21.751281 21936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:21.751340 21936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:21.751359 21936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:21.751375 21936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:21.751390 21936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:21.756615 21936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38563
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:21.757642 21936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:21.758661 21936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:21.767782 21944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:21.767876 21936 server_base.cc:1034] running on GCE node
W20260924 20:12:21.767827 21942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:21.767799 21941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:21.768589 21936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:21.769647 21936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:21.770807 21936 hybrid_clock.cc:648] HybridClock initialized: now 1790280741770768 us; error 51 us; skew 500 ppm
Sep 24 20:12:21 dist-test-slave-2pm7 krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:21.773703 21936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:12:21.774659 21936 webserver.cc:460] Webserver started at http://127.0.0.1:42109/ using document root <none> and password file <none>
I20260924 20:12:21.775230 21936 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:21.775321 21936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:21.775565 21936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:21.777942 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "7494878ac88940f690056c7cf4985b89"
format_stamp: "Formatted at 2026-09-24 20:12:21 on dist-test-slave-2pm7"
server_key: "a4539f460a29ddfd9850cdbebfae990b"
server_key_iv: "130fbe6ccf936e468a1ead040559f173"
server_key_version: "encryptionkey@0"
I20260924 20:12:21.778510 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "7494878ac88940f690056c7cf4985b89"
format_stamp: "Formatted at 2026-09-24 20:12:21 on dist-test-slave-2pm7"
server_key: "a4539f460a29ddfd9850cdbebfae990b"
server_key_iv: "130fbe6ccf936e468a1ead040559f173"
server_key_version: "encryptionkey@0"
I20260924 20:12:21.782310 21936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 20:12:21.785111 21951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:21.786454 21936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:12:21.786628 21936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "7494878ac88940f690056c7cf4985b89"
format_stamp: "Formatted at 2026-09-24 20:12:21 on dist-test-slave-2pm7"
server_key: "a4539f460a29ddfd9850cdbebfae990b"
server_key_iv: "130fbe6ccf936e468a1ead040559f173"
server_key_version: "encryptionkey@0"
I20260924 20:12:21.786762 21936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:21.791528 21936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:21.794847 21936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:21.795047 21936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:21.802584 21936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38563
I20260924 20:12:21.802604 22003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38563 every 8 connection(s)
I20260924 20:12:21.803548 21936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:21.806578 22004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:21.810722 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 21936
I20260924 20:12:21.810842 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:21.811147 462 external_mini_cluster.cc:1428] Setting key 8e79b56c2003f7d7b27ae7949584b321
I20260924 20:12:21.813421 22004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89: Bootstrap starting.
I20260924 20:12:21.816557 22004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:21.816967 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:21.812706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:38417 (local address 127.0.0.1:38563)
0924 20:12:21.813157 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:21.813182 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:21.813219 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:12:21.813936 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:21.813957 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:21.814268 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:21.814571 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:21.815528 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:21.815533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:21.815547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:21.815559 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:21.815629 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:21.815650 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:21.815766 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:21.815911 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 20:12:21.816170 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":138,"threads_started":1}
I20260924 20:12:21.817600 22004 log.cc:824] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:21.820236 22004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89: No bootstrap required, opened a new log
I20260924 20:12:21.820256 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:21.824268 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:21.822330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.59:45133 (local address 127.0.0.1:38563)
0924 20:12:21.822467 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:21.822474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:21.822715 (+ 241us) server_negotiation.cc:407] Connection header received
0924 20:12:21.822889 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:21.822892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:21.822934 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:21.823020 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:21.823848 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:21.823851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:21.823854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:21.823867 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:21.823897 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:21.823907 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:21.824030 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:21.824103 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 20:12:21.824146 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:12:21.824761 22004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494878ac88940f690056c7cf4985b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 20:12:21.825078 22004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:21.825169 22004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7494878ac88940f690056c7cf4985b89, State: Initialized, Role: FOLLOWER
I20260924 20:12:21.825719 22004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494878ac88940f690056c7cf4985b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 20:12:21.825883 22004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:21.825963 22004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:21.826098 22004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:21.827152 22004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494878ac88940f690056c7cf4985b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 20:12:21.827533 22004 leader_election.cc:302] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7494878ac88940f690056c7cf4985b89; no voters:
I20260924 20:12:21.827868 22004 leader_election.cc:288] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:21.828016 22014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:21.828265 22014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [term 1 LEADER]: Becoming Leader. State: Replica: 7494878ac88940f690056c7cf4985b89, State: Running, Role: LEADER
I20260924 20:12:21.828606 22014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494878ac88940f690056c7cf4985b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 20:12:21.829347 22004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:21.831007 22016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7494878ac88940f690056c7cf4985b89. Latest consensus state: current_term: 1 leader_uuid: "7494878ac88940f690056c7cf4985b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494878ac88940f690056c7cf4985b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } } }
I20260924 20:12:21.831140 22015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7494878ac88940f690056c7cf4985b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494878ac88940f690056c7cf4985b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } } }
I20260924 20:12:21.831302 22015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:21.831198 22016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:21.832082 22023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:21.835064 22023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:21.841074 22023 catalog_manager.cc:1347] Generated new cluster ID: d3893fff306b46da97e4f9138b9ec043
I20260924 20:12:21.841166 22023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:21.861533 22023 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:21.861871 22023 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:21.868217 22023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7494878ac88940f690056c7cf4985b89: Generated new TSK 0
I20260924 20:12:21.868981 22023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:21.880942 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 21936
2026-09-24T20:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:21 dist-test-slave-2pm7 krb5kdc[22035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:21 dist-test-slave-2pm7 krb5kdc[22035](info): set up 2 sockets
Sep 24 20:12:21 dist-test-slave-2pm7 krb5kdc[22035](info): commencing operation
krb5kdc: starting...
W20260924 20:12:22.928890 462 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:23 dist-test-slave-2pm7 krb5kdc[22035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:23.054909 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:45381
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:45381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:23.168277 22048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:23.168617 22048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:23.168704 22048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:23.172845 22048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:23.172941 22048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:23.172979 22048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:23.173038 22048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:23.173079 22048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:23.178625 22048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:45381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:45381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:23.179819 22048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:23.181084 22048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:23.189958 22054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:23.189958 22056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:23.189996 22053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:23.190203 22048 server_base.cc:1034] running on GCE node
I20260924 20:12:23.190917 22048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:23.192174 22048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:23.193390 22048 hybrid_clock.cc:648] HybridClock initialized: now 1790280743193373 us; error 28 us; skew 500 ppm
Sep 24 20:12:23 dist-test-slave-2pm7 krb5kdc[22035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:23.196388 22048 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:23.197393 22048 webserver.cc:460] Webserver started at http://127.0.115.190:40119/ using document root <none> and password file <none>
I20260924 20:12:23.198009 22048 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:23.198100 22048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:23.198343 22048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:23.200538 22048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "6ccc200c56294e08947c555c3992c96e"
format_stamp: "Formatted at 2026-09-24 20:12:23 on dist-test-slave-2pm7"
server_key: "675b2dd0113436c097c91bee7be340a9"
server_key_iv: "39aa3a58382b0abd55829aa87042914b"
server_key_version: "encryptionkey@0"
I20260924 20:12:23.201195 22048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "6ccc200c56294e08947c555c3992c96e"
format_stamp: "Formatted at 2026-09-24 20:12:23 on dist-test-slave-2pm7"
server_key: "675b2dd0113436c097c91bee7be340a9"
server_key_iv: "39aa3a58382b0abd55829aa87042914b"
server_key_version: "encryptionkey@0"
I20260924 20:12:23.205029 22048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:12:23.207486 22063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:23.208799 22048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:23.208947 22048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "6ccc200c56294e08947c555c3992c96e"
format_stamp: "Formatted at 2026-09-24 20:12:23 on dist-test-slave-2pm7"
server_key: "675b2dd0113436c097c91bee7be340a9"
server_key_iv: "39aa3a58382b0abd55829aa87042914b"
server_key_version: "encryptionkey@0"
I20260924 20:12:23.209074 22048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:23.233548 22048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:23.237059 22048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:23.237289 22048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:23.244730 22048 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:45381
I20260924 20:12:23.244746 22115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:45381 every 8 connection(s)
I20260924 20:12:23.245725 22048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:23.248703 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:23.251766 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22048
I20260924 20:12:23.251919 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:23.252270 462 external_mini_cluster.cc:1428] Setting key 4d7107fa3b1e1ceabde331c451c96a83
I20260924 20:12:23.255153 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e: Bootstrap starting.
I20260924 20:12:23.257517 22116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:23.258278 22116 log.cc:824] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:23.260386 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e: No bootstrap required, opened a new log
Sep 24 20:12:23 dist-test-slave-2pm7 krb5kdc[22035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:23.263862 22116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccc200c56294e08947c555c3992c96e" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45381 } }
I20260924 20:12:23.264226 22116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:23.264310 22116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccc200c56294e08947c555c3992c96e, State: Initialized, Role: FOLLOWER
I20260924 20:12:23.264776 22116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccc200c56294e08947c555c3992c96e" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45381 } }
I20260924 20:12:23.264935 22116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:23.265024 22116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:23.265131 22116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:23.266084 22116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccc200c56294e08947c555c3992c96e" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45381 } }
I20260924 20:12:23.266475 22116 leader_election.cc:302] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ccc200c56294e08947c555c3992c96e; no voters:
I20260924 20:12:23.266899 22116 leader_election.cc:288] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:23.267282 22122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:23.267535 22122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [term 1 LEADER]: Becoming Leader. State: Replica: 6ccc200c56294e08947c555c3992c96e, State: Running, Role: LEADER
I20260924 20:12:23.267899 22122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccc200c56294e08947c555c3992c96e" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45381 } }
I20260924 20:12:23.268179 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:23.253694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55671 (local address 127.0.115.190:45381)
0924 20:12:23.254181 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:23.254197 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:23.254232 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:12:23.254862 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:23.254879 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:23.255225 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:23.255636 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:23.256433 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:23.257437 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:23.258368 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:23.259064 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:23.259239 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:23.261880 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:23.261885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:23.261900 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:23.261928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:23.264606 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:23.265213 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:23.265220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:23.265227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:23.265365 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:23.265709 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:23.265713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:23.265715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:23.266162 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:23.266389 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:23.266774 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 20:12:23.267035 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":159,"threads_started":1}
I20260924 20:12:23.268347 22116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:23.271312 22123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ccc200c56294e08947c555c3992c96e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccc200c56294e08947c555c3992c96e" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45381 } } }
I20260924 20:12:23.271411 22123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:23.271272 22121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ccc200c56294e08947c555c3992c96e. Latest consensus state: current_term: 1 leader_uuid: "6ccc200c56294e08947c555c3992c96e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccc200c56294e08947c555c3992c96e" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45381 } } }
I20260924 20:12:23.271706 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:23.271682 22121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:23.271843 22136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:23.275643 22136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:23.282568 22136 catalog_manager.cc:1347] Generated new cluster ID: 82e99690e271458ba9ec4e41f48235ff
I20260924 20:12:23.282917 22136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:23.285518 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:23.274009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49591 (local address 127.0.115.190:45381)
0924 20:12:23.274122 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:23.274127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:23.274366 (+ 239us) server_negotiation.cc:407] Connection header received
0924 20:12:23.274583 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:23.274586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:23.274632 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:23.274686 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:23.275654 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:23.276444 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:23.277559 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:23.277865 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:23.277924 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:23.281558 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:23.281562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:23.281565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:23.281592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:23.283617 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:23.284204 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:23.284207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:23.284209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:23.284296 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:23.284634 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:23.284637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:23.284639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:23.284822 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:23.284987 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:23.285261 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 20:12:23.285356 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 20:12:23.295539 22136 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:23.296808 22136 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:23.304195 22136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ccc200c56294e08947c555c3992c96e: Generated new TSK 0
I20260924 20:12:23.304924 22136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:23.310423 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22048
W20260924 20:12:23.315922 22033 connection.cc:535] client connection to 127.0.115.190:45381 recv error: Network error: failed to read from TLS socket (remote: 127.0.115.190:45381): Connection reset by peer (error 104)
2026-09-24T20:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:23 dist-test-slave-2pm7 krb5kdc[22147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:23 dist-test-slave-2pm7 krb5kdc[22147](info): set up 2 sockets
Sep 24 20:12:23 dist-test-slave-2pm7 krb5kdc[22147](info): commencing operation
krb5kdc: starting...
W20260924 20:12:24.356437 462 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:24 dist-test-slave-2pm7 krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:24.483191 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46489
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:24.598912 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:24.599191 22160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:24.599233 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:24.603318 22160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:24.603389 22160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:24.603407 22160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:24.603425 22160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:24.603439 22160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:24.608771 22160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:46489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:24.609838 22160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:24.610880 22160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:24.619529 22166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:24.619570 22168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:24.619676 22160 server_base.cc:1034] running on GCE node
W20260924 20:12:24.619868 22165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:24.620394 22160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:24.621577 22160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:24.622786 22160 hybrid_clock.cc:648] HybridClock initialized: now 1790280744622772 us; error 25 us; skew 500 ppm
Sep 24 20:12:24 dist-test-slave-2pm7 krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:24.625782 22160 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:24.626725 22160 webserver.cc:460] Webserver started at http://127.0.115.190:46713/ using document root <none> and password file <none>
I20260924 20:12:24.627307 22160 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:24.627410 22160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:24.627647 22160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:24.629803 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3"
format_stamp: "Formatted at 2026-09-24 20:12:24 on dist-test-slave-2pm7"
server_key: "dcc68d12aeb4eb6c5434bbaae38aebad"
server_key_iv: "a8b207d9b1583c324a3af8386e37834b"
server_key_version: "encryptionkey@0"
I20260924 20:12:24.630369 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3"
format_stamp: "Formatted at 2026-09-24 20:12:24 on dist-test-slave-2pm7"
server_key: "dcc68d12aeb4eb6c5434bbaae38aebad"
server_key_iv: "a8b207d9b1583c324a3af8386e37834b"
server_key_version: "encryptionkey@0"
I20260924 20:12:24.634112 22160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:12:24.636502 22175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:24.637735 22160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:24.637876 22160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3"
format_stamp: "Formatted at 2026-09-24 20:12:24 on dist-test-slave-2pm7"
server_key: "dcc68d12aeb4eb6c5434bbaae38aebad"
server_key_iv: "a8b207d9b1583c324a3af8386e37834b"
server_key_version: "encryptionkey@0"
I20260924 20:12:24.637984 22160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:24.649662 22160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:24.652436 22160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:24.652628 22160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:24.659865 22160 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:46489
I20260924 20:12:24.659890 22227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:46489 every 8 connection(s)
I20260924 20:12:24.660822 22160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:24.663466 22228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:24.668907 22228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3: Bootstrap starting.
I20260924 20:12:24.669566 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22160
I20260924 20:12:24.669701 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:24.669981 462 external_mini_cluster.cc:1428] Setting key f6eca738849ec1467e1e9180c9a0c187
I20260924 20:12:24.671890 22228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:24.672647 22228 log.cc:824] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:24.674580 22228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3: No bootstrap required, opened a new log
I20260924 20:12:24.678177 22228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46489 } }
I20260924 20:12:24.678489 22228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:24.678570 22228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f042d3ea2d4c65bc45f3f7cde9b7a3, State: Initialized, Role: FOLLOWER
I20260924 20:12:24.679011 22228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46489 } }
I20260924 20:12:24.679128 22228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:24.679211 22228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:24.679301 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:24.671364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45189 (local address 127.0.115.190:46489)
0924 20:12:24.671799 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:24.671814 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:24.671848 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:12:24.672587 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:24.672607 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:24.672998 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:24.673337 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:24.673772 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:24.674799 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:24.675700 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:24.676242 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:24.676413 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:24.677374 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:24.677379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:24.677391 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:24.677780 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:24.677831 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:24.677854 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:24.677995 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:24.678183 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 20:12:24.678318 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260924 20:12:24.679323 22228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:24.680330 22228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46489 } }
I20260924 20:12:24.680681 22228 leader_election.cc:302] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07f042d3ea2d4c65bc45f3f7cde9b7a3; no voters:
I20260924 20:12:24.680989 22228 leader_election.cc:288] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:24.681061 22233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:24.681288 22233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [term 1 LEADER]: Becoming Leader. State: Replica: 07f042d3ea2d4c65bc45f3f7cde9b7a3, State: Running, Role: LEADER
I20260924 20:12:24.681609 22233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46489 } }
I20260924 20:12:24.682274 22228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:24.682279 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:24.684192 22235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f042d3ea2d4c65bc45f3f7cde9b7a3. Latest consensus state: current_term: 1 leader_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46489 } } }
I20260924 20:12:24.684337 22235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:24.687569 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f042d3ea2d4c65bc45f3f7cde9b7a3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 46489 } } }
I20260924 20:12:24.687703 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3 [sys.catalog]: This master's current role is: LEADER
W20260924 20:12:24.688180 22251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:24.688262 22251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:24.688342 22248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:24.689014 22248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:24.692359 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:24.685866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42001 (local address 127.0.115.190:46489)
0924 20:12:24.686021 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:24.686025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:24.686591 (+ 566us) server_negotiation.cc:407] Connection header received
0924 20:12:24.686799 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:24.686803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:24.686851 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:24.686902 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:24.687374 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:24.688199 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:24.689398 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:24.689719 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:24.689834 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:24.690794 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:24.690798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:24.690801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:24.690814 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:24.690842 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:24.690855 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:24.690953 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:24.692187 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0924 20:12:24.692222 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 20:12:24.695372 22248 catalog_manager.cc:1347] Generated new cluster ID: c0d604a5e28c4deb9a88c554d05b9db7
I20260924 20:12:24.695446 22248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:24.705852 22248 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:24.706751 22248 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:24.712368 22248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07f042d3ea2d4c65bc45f3f7cde9b7a3: Generated new TSK 0
I20260924 20:12:24.713119 22248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:24.724781 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22160
W20260924 20:12:24.730877 22145 connection.cc:535] client connection to 127.0.115.190:46489 recv error: Network error: failed to read from TLS socket (remote: 127.0.115.190:46489): Connection reset by peer (error 104)
2026-09-24T20:12:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:24 dist-test-slave-2pm7 krb5kdc[22259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:24 dist-test-slave-2pm7 krb5kdc[22259](info): set up 2 sockets
Sep 24 20:12:24 dist-test-slave-2pm7 krb5kdc[22259](info): commencing operation
krb5kdc: starting...
W20260924 20:12:25.752348 462 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:25 dist-test-slave-2pm7 krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:25.878993 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44069
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:44069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:25.993209 22272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:25.993525 22272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:25.993643 22272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:25.997799 22272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:25.997890 22272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:25.997925 22272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:25.997964 22272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:25.997994 22272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:26.003263 22272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:44069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:26.004472 22272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:26.005609 22272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:26.014429 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:26.014487 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:26.014640 22272 server_base.cc:1034] running on GCE node
W20260924 20:12:26.014422 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:26.015302 22272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:26.016541 22272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:26.017704 22272 hybrid_clock.cc:648] HybridClock initialized: now 1790280746017673 us; error 44 us; skew 500 ppm
Sep 24 20:12:26 dist-test-slave-2pm7 krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:26.024936 22272 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:26.025950 22272 webserver.cc:460] Webserver started at http://127.0.115.190:38015/ using document root <none> and password file <none>
I20260924 20:12:26.026549 22272 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:26.026643 22272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:26.026878 22272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:26.029050 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "ebdc52d21de740be90ca014b44cda1c0"
format_stamp: "Formatted at 2026-09-24 20:12:26 on dist-test-slave-2pm7"
server_key: "17322b14887120855e0a0dfcbbe1a6e6"
server_key_iv: "dad9933c1fbca9992783fc4421bdffd6"
server_key_version: "encryptionkey@0"
I20260924 20:12:26.029624 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "ebdc52d21de740be90ca014b44cda1c0"
format_stamp: "Formatted at 2026-09-24 20:12:26 on dist-test-slave-2pm7"
server_key: "17322b14887120855e0a0dfcbbe1a6e6"
server_key_iv: "dad9933c1fbca9992783fc4421bdffd6"
server_key_version: "encryptionkey@0"
I20260924 20:12:26.033432 22272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:12:26.035792 22287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:26.037117 22272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:26.037273 22272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "ebdc52d21de740be90ca014b44cda1c0"
format_stamp: "Formatted at 2026-09-24 20:12:26 on dist-test-slave-2pm7"
server_key: "17322b14887120855e0a0dfcbbe1a6e6"
server_key_iv: "dad9933c1fbca9992783fc4421bdffd6"
server_key_version: "encryptionkey@0"
I20260924 20:12:26.037388 22272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:26.041975 22272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:26.045110 22272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:26.045324 22272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:26.052843 22272 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:44069
I20260924 20:12:26.052870 22339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:44069 every 8 connection(s)
I20260924 20:12:26.053817 22272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:26.055928 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22272
I20260924 20:12:26.056100 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:26.056403 462 external_mini_cluster.cc:1428] Setting key 3d18013ea25b0aaf742027d691cb8ccc
I20260924 20:12:26.056936 22340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:26.062321 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:26.057886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52493 (local address 127.0.115.190:44069)
0924 20:12:26.058328 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:26.058351 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:26.058396 (+ 45us) server_negotiation.cc:407] Connection header received
0924 20:12:26.059218 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:26.059244 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:26.059625 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:26.060058 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:26.060820 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:26.060827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:26.060844 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:26.060857 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:26.060924 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:26.060953 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:26.061147 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:26.061221 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 20:12:26.061376 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260924 20:12:26.064462 22340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0: Bootstrap starting.
I20260924 20:12:26.065348 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:26.066973 22340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:26.067894 22340 log.cc:824] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:26.069195 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:26.067374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59477 (local address 127.0.115.190:44069)
0924 20:12:26.067511 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:26.067517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:26.067723 (+ 206us) server_negotiation.cc:407] Connection header received
0924 20:12:26.067850 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:26.067853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:26.067882 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:26.067998 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:26.068782 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:26.068790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:26.068792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:26.068802 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:26.068826 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:26.068836 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:26.068904 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:26.069019 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 20:12:26.069049 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:12:26.070330 22340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0: No bootstrap required, opened a new log
I20260924 20:12:26.074074 22340 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc52d21de740be90ca014b44cda1c0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44069 } }
I20260924 20:12:26.074337 22340 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:26.074419 22340 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdc52d21de740be90ca014b44cda1c0, State: Initialized, Role: FOLLOWER
I20260924 20:12:26.074898 22340 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc52d21de740be90ca014b44cda1c0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44069 } }
I20260924 20:12:26.075062 22340 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:26.075157 22340 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:26.075263 22340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:26.076555 22340 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc52d21de740be90ca014b44cda1c0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44069 } }
I20260924 20:12:26.076921 22340 leader_election.cc:302] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebdc52d21de740be90ca014b44cda1c0; no voters:
I20260924 20:12:26.077255 22340 leader_election.cc:288] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:26.077366 22350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:26.077656 22350 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [term 1 LEADER]: Becoming Leader. State: Replica: ebdc52d21de740be90ca014b44cda1c0, State: Running, Role: LEADER
I20260924 20:12:26.077963 22350 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc52d21de740be90ca014b44cda1c0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44069 } }
I20260924 20:12:26.078555 22340 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:26.079140 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebdc52d21de740be90ca014b44cda1c0. Latest consensus state: current_term: 1 leader_uuid: "ebdc52d21de740be90ca014b44cda1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc52d21de740be90ca014b44cda1c0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44069 } } }
I20260924 20:12:26.079314 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:26.079607 22352 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebdc52d21de740be90ca014b44cda1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc52d21de740be90ca014b44cda1c0" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 44069 } } }
I20260924 20:12:26.079718 22352 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:26.079742 22357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:26.083077 22357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:26.089190 22357 catalog_manager.cc:1347] Generated new cluster ID: fea69eb54cd045238e270b4810f0d52e
I20260924 20:12:26.089265 22357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:26.097318 22357 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:26.097636 22357 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:26.104715 22357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebdc52d21de740be90ca014b44cda1c0: Generated new TSK 0
I20260924 20:12:26.105369 22357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:26.109963 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22272
2026-09-24T20:12:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:26 dist-test-slave-2pm7 krb5kdc[22371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:26 dist-test-slave-2pm7 krb5kdc[22371](info): set up 2 sockets
Sep 24 20:12:26 dist-test-slave-2pm7 krb5kdc[22371](info): commencing operation
krb5kdc: starting...
W20260924 20:12:27.156512 462 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:27 dist-test-slave-2pm7 krb5kdc[22371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:27.280944 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:45547
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:45547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:27.392899 22384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:27.393214 22384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:27.393291 22384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:27.397368 22384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:27.397460 22384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:27.397497 22384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:27.397552 22384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:27.397593 22384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:27.403074 22384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:45547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:45547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:27.404345 22384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:27.405443 22384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:27.414196 22389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:27.414196 22390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:27.414278 22384 server_base.cc:1034] running on GCE node
W20260924 20:12:27.414266 22392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:27.415046 22384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:27.416213 22384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:27.417397 22384 hybrid_clock.cc:648] HybridClock initialized: now 1790280747417375 us; error 41 us; skew 500 ppm
Sep 24 20:12:27 dist-test-slave-2pm7 krb5kdc[22371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:27.420094 22384 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:27.421011 22384 webserver.cc:460] Webserver started at http://127.0.115.190:43929/ using document root <none> and password file <none>
I20260924 20:12:27.421574 22384 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:27.421669 22384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:27.421900 22384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:27.424077 22384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "826e661c2c8c45a1af82df3ad105f3e5"
format_stamp: "Formatted at 2026-09-24 20:12:27 on dist-test-slave-2pm7"
server_key: "1d436750755ad67e10fd6329a9e896fb"
server_key_iv: "84961507a2548f6dc8f1cb29e2c2d7d3"
server_key_version: "encryptionkey@0"
I20260924 20:12:27.424669 22384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "826e661c2c8c45a1af82df3ad105f3e5"
format_stamp: "Formatted at 2026-09-24 20:12:27 on dist-test-slave-2pm7"
server_key: "1d436750755ad67e10fd6329a9e896fb"
server_key_iv: "84961507a2548f6dc8f1cb29e2c2d7d3"
server_key_version: "encryptionkey@0"
I20260924 20:12:27.428375 22384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:12:27.430728 22399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:27.431988 22384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:27.432133 22384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "826e661c2c8c45a1af82df3ad105f3e5"
format_stamp: "Formatted at 2026-09-24 20:12:27 on dist-test-slave-2pm7"
server_key: "1d436750755ad67e10fd6329a9e896fb"
server_key_iv: "84961507a2548f6dc8f1cb29e2c2d7d3"
server_key_version: "encryptionkey@0"
I20260924 20:12:27.432240 22384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:27.454146 22384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:27.457203 22384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:27.457418 22384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:27.464788 22384 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:45547
I20260924 20:12:27.464810 22451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:45547 every 8 connection(s)
I20260924 20:12:27.465822 22384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:27.467374 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22384
I20260924 20:12:27.467526 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:27.467864 462 external_mini_cluster.cc:1428] Setting key 37694d7a5f70fc543ad7490383c2bcd1
I20260924 20:12:27.469007 22452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:27.474726 22452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5: Bootstrap starting.
Sep 24 20:12:27 dist-test-slave-2pm7 krb5kdc[22371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:27.477041 22452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:27.477896 22452 log.cc:824] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:27.480041 22452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5: No bootstrap required, opened a new log
I20260924 20:12:27.482002 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:27.469343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41965 (local address 127.0.115.190:45547)
0924 20:12:27.469823 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:27.469838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:27.469876 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:12:27.470549 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:27.470566 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:27.470919 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:27.471266 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:27.472142 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:27.473157 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:27.474012 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:27.474498 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:27.474672 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:27.477213 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:27.477219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:27.477232 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:27.477261 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:27.479096 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:27.479529 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:27.479535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:27.479540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:27.479666 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:27.479988 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:27.479990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:27.479992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:27.480420 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:27.480636 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:27.480906 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 20:12:27.481195 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":155,"threads_started":1}
I20260924 20:12:27.484545 22452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826e661c2c8c45a1af82df3ad105f3e5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45547 } }
I20260924 20:12:27.484829 22452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:27.484946 22452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826e661c2c8c45a1af82df3ad105f3e5, State: Initialized, Role: FOLLOWER
I20260924 20:12:27.485457 22452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826e661c2c8c45a1af82df3ad105f3e5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45547 } }
I20260924 20:12:27.485622 22452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:27.485739 22452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:27.485814 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:27.485863 22452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:27.487115 22452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826e661c2c8c45a1af82df3ad105f3e5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45547 } }
I20260924 20:12:27.487540 22452 leader_election.cc:302] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 826e661c2c8c45a1af82df3ad105f3e5; no voters:
I20260924 20:12:27.487982 22452 leader_election.cc:288] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:27.488049 22461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:27.488258 22461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [term 1 LEADER]: Becoming Leader. State: Replica: 826e661c2c8c45a1af82df3ad105f3e5, State: Running, Role: LEADER
I20260924 20:12:27.488567 22461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826e661c2c8c45a1af82df3ad105f3e5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45547 } }
I20260924 20:12:27.489418 22452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:27.489820 22461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 826e661c2c8c45a1af82df3ad105f3e5. Latest consensus state: current_term: 1 leader_uuid: "826e661c2c8c45a1af82df3ad105f3e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826e661c2c8c45a1af82df3ad105f3e5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45547 } } }
I20260924 20:12:27.489936 22461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:27.490126 22461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "826e661c2c8c45a1af82df3ad105f3e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826e661c2c8c45a1af82df3ad105f3e5" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 45547 } } }
I20260924 20:12:27.490223 22461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:27.490407 22466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:27.493695 22466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:27.499737 22466 catalog_manager.cc:1347] Generated new cluster ID: b8a0ab79dc674688a71a641a5246d98e
I20260924 20:12:27.499797 22466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:27.501708 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:27.488169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51021 (local address 127.0.115.190:45547)
0924 20:12:27.489656 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:27.489663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:27.491778 (+ 2115us) server_negotiation.cc:407] Connection header received
0924 20:12:27.492005 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:27.492008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:27.492059 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:27.492134 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:27.493148 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:27.493984 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:27.494904 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:27.495179 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:27.495291 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:27.498444 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:27.498448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:27.498451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:27.498476 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:27.500071 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:27.500500 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:27.500504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:27.500506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:27.500586 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:27.500849 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:27.500853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:27.500855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:27.501034 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:27.501184 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:27.501414 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 20:12:27.501530 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241}
I20260924 20:12:27.512276 22466 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:27.513545 22466 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:27.527846 22466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 826e661c2c8c45a1af82df3ad105f3e5: Generated new TSK 0
I20260924 20:12:27.528661 22466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:27.539883 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22384
2026-09-24T20:12:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:27 dist-test-slave-2pm7 krb5kdc[22483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:27 dist-test-slave-2pm7 krb5kdc[22483](info): set up 2 sockets
Sep 24 20:12:27 dist-test-slave-2pm7 krb5kdc[22483](info): commencing operation
krb5kdc: starting...
W20260924 20:12:28.572964 462 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:28 dist-test-slave-2pm7 krb5kdc[22483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:28.698933 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:37333
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:28.809947 22496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:28.810230 22496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:28.810272 22496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:28.814366 22496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:28.814436 22496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:28.814471 22496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:28.814489 22496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:28.814504 22496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:28.819901 22496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:37333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:28.821060 22496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:28.822196 22496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:28.831095 22502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:28.831104 22501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:28.831175 22504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:28.831390 22496 server_base.cc:1034] running on GCE node
I20260924 20:12:28.832168 22496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:28.833395 22496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:28.834595 22496 hybrid_clock.cc:648] HybridClock initialized: now 1790280748834569 us; error 43 us; skew 500 ppm
Sep 24 20:12:28 dist-test-slave-2pm7 krb5kdc[22483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:28.837361 22496 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:28.838346 22496 webserver.cc:460] Webserver started at http://127.0.115.190:41807/ using document root <none> and password file <none>
I20260924 20:12:28.838917 22496 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:28.839003 22496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:28.839231 22496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:28.841364 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "a28cc78c5d6c411baa25f0be87d7b53c"
format_stamp: "Formatted at 2026-09-24 20:12:28 on dist-test-slave-2pm7"
server_key: "cc56cb35cd09ae01633673108bc5056f"
server_key_iv: "2123454d4ed4da3ba350628838133f2b"
server_key_version: "encryptionkey@0"
I20260924 20:12:28.841943 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "a28cc78c5d6c411baa25f0be87d7b53c"
format_stamp: "Formatted at 2026-09-24 20:12:28 on dist-test-slave-2pm7"
server_key: "cc56cb35cd09ae01633673108bc5056f"
server_key_iv: "2123454d4ed4da3ba350628838133f2b"
server_key_version: "encryptionkey@0"
I20260924 20:12:28.845821 22496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:12:28.848237 22511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:28.849515 22496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:12:28.849673 22496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "a28cc78c5d6c411baa25f0be87d7b53c"
format_stamp: "Formatted at 2026-09-24 20:12:28 on dist-test-slave-2pm7"
server_key: "cc56cb35cd09ae01633673108bc5056f"
server_key_iv: "2123454d4ed4da3ba350628838133f2b"
server_key_version: "encryptionkey@0"
I20260924 20:12:28.849793 22496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:28.866240 22496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:28.869289 22496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:28.869498 22496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:28.876931 22496 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:37333
I20260924 20:12:28.876958 22563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:37333 every 8 connection(s)
I20260924 20:12:28.877916 22496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:28.880668 22564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:28.885979 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22496
I20260924 20:12:28.886108 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:28.886396 462 external_mini_cluster.cc:1428] Setting key e67ce11fe723842b491c593aa1ef2f45
I20260924 20:12:28.886639 22564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c: Bootstrap starting.
I20260924 20:12:28.889137 22564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:28.889981 22564 log.cc:824] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:28.892177 22564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c: No bootstrap required, opened a new log
I20260924 20:12:28.895777 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:28.887874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41921 (local address 127.0.115.190:37333)
0924 20:12:28.888347 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:28.888363 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:28.888399 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:12:28.889035 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:28.889060 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:28.889455 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:28.889781 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:28.890161 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:28.891121 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:28.892426 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:28.892969 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:28.893114 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:28.893963 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:28.893968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:28.893983 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:28.894390 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:28.894446 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:28.894482 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:28.894566 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:28.894738 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 20:12:28.894974 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":155,"threads_started":1}
I20260924 20:12:28.896637 22564 raft_consensus.cc:348] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37333 } }
I20260924 20:12:28.896911 22564 raft_consensus.cc:374] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:28.896991 22564 raft_consensus.cc:729] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28cc78c5d6c411baa25f0be87d7b53c, State: Initialized, Role: FOLLOWER
I20260924 20:12:28.897469 22564 consensus_queue.cc:260] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37333 } }
I20260924 20:12:28.897619 22564 raft_consensus.cc:388] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:28.897688 22564 raft_consensus.cc:482] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:28.897794 22564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:28.898911 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:28.898911 22564 raft_consensus.cc:504] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37333 } }
I20260924 20:12:28.899245 22564 leader_election.cc:302] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a28cc78c5d6c411baa25f0be87d7b53c; no voters:
I20260924 20:12:28.899610 22564 leader_election.cc:288] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:28.899689 22569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:28.899883 22569 raft_consensus.cc:686] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [term 1 LEADER]: Becoming Leader. State: Replica: a28cc78c5d6c411baa25f0be87d7b53c, State: Running, Role: LEADER
I20260924 20:12:28.900254 22569 consensus_queue.cc:237] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37333 } }
I20260924 20:12:28.900986 22564 sys_catalog.cc:564] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:28.901573 22569 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a28cc78c5d6c411baa25f0be87d7b53c. Latest consensus state: current_term: 1 leader_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37333 } } }
I20260924 20:12:28.901760 22569 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:28.902035 22570 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cc78c5d6c411baa25f0be87d7b53c" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 37333 } } }
I20260924 20:12:28.902155 22570 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:28.902207 22581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:28.905702 22581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:28.906517 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:28.901096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37631 (local address 127.0.115.190:37333)
0924 20:12:28.901197 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:28.901208 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:28.901717 (+ 509us) server_negotiation.cc:407] Connection header received
0924 20:12:28.901939 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:28.901944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:28.902002 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:28.902059 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:28.902494 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:28.903344 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:28.904460 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:28.904772 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:28.904893 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:28.905751 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:28.905755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:28.905758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:28.905772 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:28.905803 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:28.905816 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:28.905897 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:28.906325 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 20:12:28.906389 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 20:12:28.911739 22581 catalog_manager.cc:1347] Generated new cluster ID: 769e803c91ed4df5a9e2ea24c6b7db8d
I20260924 20:12:28.911813 22581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:28.931627 22581 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:28.932724 22581 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:28.950596 22581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a28cc78c5d6c411baa25f0be87d7b53c: Generated new TSK 0
I20260924 20:12:28.951498 22581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:28.967504 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22496
2026-09-24T20:12:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:29 dist-test-slave-2pm7 krb5kdc[22595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:29 dist-test-slave-2pm7 krb5kdc[22595](info): set up 2 sockets
Sep 24 20:12:29 dist-test-slave-2pm7 krb5kdc[22595](info): commencing operation
krb5kdc: starting...
W20260924 20:12:30.032469 462 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:30 dist-test-slave-2pm7 krb5kdc[22595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:30.154917 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:40479
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:40479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:30.267277 22608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:30.267573 22608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:30.267620 22608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:30.271589 22608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:30.271678 22608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:30.271700 22608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:30.271716 22608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:30.271731 22608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:30.276985 22608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:40479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:40479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:30.278000 22608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:30.278977 22608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:30.287217 22614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:30.287230 22613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:30.287227 22616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:30.287449 22608 server_base.cc:1034] running on GCE node
I20260924 20:12:30.288175 22608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:30.289270 22608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:30.290453 22608 hybrid_clock.cc:648] HybridClock initialized: now 1790280750290439 us; error 25 us; skew 500 ppm
Sep 24 20:12:30 dist-test-slave-2pm7 krb5kdc[22595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:30.293277 22608 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:30.294166 22608 webserver.cc:460] Webserver started at http://127.0.115.190:39321/ using document root <none> and password file <none>
I20260924 20:12:30.294740 22608 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:30.294829 22608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:30.295049 22608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:30.297248 22608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "320a53281e8c492e81c83fe468ae8ce3"
format_stamp: "Formatted at 2026-09-24 20:12:30 on dist-test-slave-2pm7"
server_key: "3f499a0ed947d3da9220e5e733c1c1ba"
server_key_iv: "6ffb7bb5d76476a1e71069690db7e6ac"
server_key_version: "encryptionkey@0"
I20260924 20:12:30.297802 22608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "320a53281e8c492e81c83fe468ae8ce3"
format_stamp: "Formatted at 2026-09-24 20:12:30 on dist-test-slave-2pm7"
server_key: "3f499a0ed947d3da9220e5e733c1c1ba"
server_key_iv: "6ffb7bb5d76476a1e71069690db7e6ac"
server_key_version: "encryptionkey@0"
I20260924 20:12:30.301440 22608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:12:30.303867 22623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:30.305274 22608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:30.305425 22608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "320a53281e8c492e81c83fe468ae8ce3"
format_stamp: "Formatted at 2026-09-24 20:12:30 on dist-test-slave-2pm7"
server_key: "3f499a0ed947d3da9220e5e733c1c1ba"
server_key_iv: "6ffb7bb5d76476a1e71069690db7e6ac"
server_key_version: "encryptionkey@0"
I20260924 20:12:30.305540 22608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:30.310217 22608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:30.313476 22608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:30.313668 22608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:30.320788 22608 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:40479
I20260924 20:12:30.320829 22675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:40479 every 8 connection(s)
I20260924 20:12:30.321735 22608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:30.324536 22676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:30.330137 22676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3: Bootstrap starting.
I20260924 20:12:30.331009 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22608
I20260924 20:12:30.331153 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:30.331432 462 external_mini_cluster.cc:1428] Setting key 1563b024f36df9f0b80acfcd19ebeb90
I20260924 20:12:30.333474 22676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:30.334410 22676 log.cc:824] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:30.336978 22676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3: No bootstrap required, opened a new log
I20260924 20:12:30.337445 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:30.332926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50125 (local address 127.0.115.190:40479)
0924 20:12:30.333423 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:30.333439 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:30.333482 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:12:30.334303 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:30.334324 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:30.334732 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:30.335144 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:30.336030 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:30.336035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:30.336048 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:30.336059 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:30.336113 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:30.336134 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:30.336236 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:30.336409 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 20:12:30.336603 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260924 20:12:30.340736 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:30.341400 22676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a53281e8c492e81c83fe468ae8ce3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40479 } }
I20260924 20:12:30.341723 22676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:12:30.341789 22676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320a53281e8c492e81c83fe468ae8ce3, State: Initialized, Role: FOLLOWER
I20260924 20:12:30.342303 22676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a53281e8c492e81c83fe468ae8ce3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40479 } }
I20260924 20:12:30.342473 22676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:12:30.342562 22676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:12:30.342666 22676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:30.343933 22676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a53281e8c492e81c83fe468ae8ce3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40479 } }
I20260924 20:12:30.344331 22676 leader_election.cc:302] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320a53281e8c492e81c83fe468ae8ce3; no voters:
I20260924 20:12:30.344688 22676 leader_election.cc:288] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:12:30.344779 22686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:30.344919 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:30.343141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44607 (local address 127.0.115.190:40479)
0924 20:12:30.343258 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:30.343266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:30.343493 (+ 227us) server_negotiation.cc:407] Connection header received
0924 20:12:30.343581 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:30.343587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:30.343632 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:30.343719 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:30.344454 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:30.344457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:30.344460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:12:30.344472 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:30.344501 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:12:30.344514 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:30.344610 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:30.344700 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 20:12:30.344745 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:12:30.345084 22686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [term 1 LEADER]: Becoming Leader. State: Replica: 320a53281e8c492e81c83fe468ae8ce3, State: Running, Role: LEADER
I20260924 20:12:30.345388 22686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a53281e8c492e81c83fe468ae8ce3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40479 } }
I20260924 20:12:30.346266 22676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:30.347203 22688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320a53281e8c492e81c83fe468ae8ce3. Latest consensus state: current_term: 1 leader_uuid: "320a53281e8c492e81c83fe468ae8ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a53281e8c492e81c83fe468ae8ce3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40479 } } }
I20260924 20:12:30.347182 22687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320a53281e8c492e81c83fe468ae8ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a53281e8c492e81c83fe468ae8ce3" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 40479 } } }
I20260924 20:12:30.347393 22687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:30.347393 22688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:30.347834 22695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:30.351177 22695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:30.357576 22695 catalog_manager.cc:1347] Generated new cluster ID: 7139d173a26c48adae20d03b89011a0a
I20260924 20:12:30.357719 22695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:30.375422 22695 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:30.375787 22695 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:30.382439 22695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320a53281e8c492e81c83fe468ae8ce3: Generated new TSK 0
I20260924 20:12:30.383324 22695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:12:30.387224 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22608
2026-09-24T20:12:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1419 ms)
[----------] 18 tests from AuthTokenIssuingTest (25668 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:30 dist-test-slave-2pm7 krb5kdc[22707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:30 dist-test-slave-2pm7 krb5kdc[22707](info): set up 2 sockets
Sep 24 20:12:30 dist-test-slave-2pm7 krb5kdc[22707](info): commencing operation
krb5kdc: starting...
W20260924 20:12:31.421109 462 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:31 dist-test-slave-2pm7 krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:31.553130 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:42687
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:42687,127.0.115.189:32931,127.0.115.188:46469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:31.666147 22720 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:12:31.666440 22720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:31.666464 22720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:31.666503 22720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:31.670444 22720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:31.670501 22720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:31.670521 22720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:31.670537 22720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:31.670552 22720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:31.675745 22720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:42687,127.0.115.189:32931,127.0.115.188:46469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:31.676836 22720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:31.677850 22720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:31.686053 22725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:31.686090 22726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:31.686252 22720 server_base.cc:1034] running on GCE node
W20260924 20:12:31.686215 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:31.686961 22720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:31.688256 22720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:31.689437 22720 hybrid_clock.cc:648] HybridClock initialized: now 1790280751689403 us; error 45 us; skew 500 ppm
Sep 24 20:12:31 dist-test-slave-2pm7 krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:31.697063 22720 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:31.697992 22720 webserver.cc:460] Webserver started at http://127.0.115.190:44469/ using document root <none> and password file <none>
I20260924 20:12:31.698551 22720 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:31.698637 22720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:31.698868 22720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:31.701098 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "b0e8e4ebe30b47aea03a8850317ff8f2"
format_stamp: "Formatted at 2026-09-24 20:12:31 on dist-test-slave-2pm7"
server_key: "a77eff36ff6ade90cb70f8ce626bf912"
server_key_iv: "f3df68b2c39bf99e1152fa688e52044b"
server_key_version: "encryptionkey@0"
I20260924 20:12:31.701639 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "b0e8e4ebe30b47aea03a8850317ff8f2"
format_stamp: "Formatted at 2026-09-24 20:12:31 on dist-test-slave-2pm7"
server_key: "a77eff36ff6ade90cb70f8ce626bf912"
server_key_iv: "f3df68b2c39bf99e1152fa688e52044b"
server_key_version: "encryptionkey@0"
I20260924 20:12:31.705418 22720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:12:31.707872 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:31.709147 22720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:31.709308 22720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "b0e8e4ebe30b47aea03a8850317ff8f2"
format_stamp: "Formatted at 2026-09-24 20:12:31 on dist-test-slave-2pm7"
server_key: "a77eff36ff6ade90cb70f8ce626bf912"
server_key_iv: "f3df68b2c39bf99e1152fa688e52044b"
server_key_version: "encryptionkey@0"
I20260924 20:12:31.709409 22720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:31.742883 22720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:31.745903 22720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:31.746068 22720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:31.753263 22720 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:42687
I20260924 20:12:31.753548 22787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:42687 every 8 connection(s)
I20260924 20:12:31.754324 22720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:31.757184 22788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:31.760146 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22720
I20260924 20:12:31.760272 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:31.760542 462 external_mini_cluster.cc:1428] Setting key 8d54d51cd540f4bae15ad2e44841d338
I20260924 20:12:31.761137 22788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.115.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.189@KRBTEST.COM" created.
Sep 24 20:12:31 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:31.777070 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.763456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.0.115.190:42687)
0924 20:12:31.763876 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:31.763896 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:31.764191 (+ 295us) server_negotiation.cc:407] Connection header received
0924 20:12:31.765204 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:31.765208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:31.765383 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:31.765565 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:31.766392 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:31.767260 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:31.768341 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:31.768707 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:31.768876 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:31.771984 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:31.771994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:31.772006 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:31.772032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:31.773958 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:31.774406 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:31.774412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:31.774414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:31.774528 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:31.774836 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:31.774839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:31.774841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:31.775228 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:31.775361 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:31.775731 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 20:12:31.776003 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":273,"thread_start_us":133,"threads_started":1}
I20260924 20:12:31.777070 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.763457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:42687 (local address 127.0.0.1:58974)
0924 20:12:31.764045 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.764084 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:31.764904 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:31.765565 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:31.765570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:31.766010 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:31.766249 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:31.766262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:31.767393 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:31.767395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:31.768201 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:31.768211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:31.768860 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:31.768864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:31.768983 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:31.769602 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:31.769620 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:31.771806 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:31.774058 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:31.774068 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:31.774079 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:31.774299 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:31.774631 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:31.774634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:31.774636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:31.774740 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:31.775344 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:31.775350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:31.775485 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 20:12:31.775666 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 20:12:31.776057 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":126,"spinlock_wait_cycles":39680,"thread_start_us":176,"threads_started":1}
I20260924 20:12:31.779515 22788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:31.780328 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.780009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56790)
0924 20:12:31.780115 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.780236 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 20:12:31.780514 22739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
I20260924 20:12:31.781314 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.781102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56796)
0924 20:12:31.781212 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.781260 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:12:31.781488 22788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:32931: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab.
Entry for principal kudu/127.0.115.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab.
I20260924 20:12:31.787607 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:32931
--webserver_interface=127.0.115.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab
--principal=kudu/127.0.115.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:42687,127.0.115.189:32931,127.0.115.188:46469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:12:31.830789 22788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } attempt: 1
I20260924 20:12:31.831749 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.831392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56804)
0924 20:12:31.831556 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.831668 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 20:12:31.832541 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.832322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56810)
0924 20:12:31.832435 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.832486 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:12:31.832696 22788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:32931: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
I20260924 20:12:31.900852 22788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } attempt: 2
I20260924 20:12:31.901660 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.901370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56818)
0924 20:12:31.901510 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.901577 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 20:12:31.902401 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.902215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56834)
0924 20:12:31.902308 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.902351 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 20:12:31.902561 22788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:32931: Network error: Client connection negotiation failed: client connection to 127.0.115.189:32931: connect: Connection refused (error 111)
W20260924 20:12:31.902962 22795 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:12:31.903309 22795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:31.903352 22795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:31.903431 22795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:31.908192 22795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:31.908262 22795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:31.908282 22795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:31.908330 22795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:31.908358 22795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:31.914150 22795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:42687,127.0.115.189:32931,127.0.115.188:46469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab
--principal=kudu/127.0.115.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:32931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.115.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:31.915382 22795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:31.916518 22795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:31.925104 22803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:31.925155 22801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:31.925104 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:31.925382 22795 server_base.cc:1034] running on GCE node
I20260924 20:12:31.926034 22795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:31.927254 22795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:31.928469 22795 hybrid_clock.cc:648] HybridClock initialized: now 1790280751928442 us; error 41 us; skew 500 ppm
Sep 24 20:12:31 dist-test-slave-2pm7 krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:31.931277 22795 init.cc:377] Logged in from keytab as kudu/127.0.115.189@KRBTEST.COM (short username kudu)
I20260924 20:12:31.932300 22795 webserver.cc:460] Webserver started at http://127.0.115.189:42009/ using document root <none> and password file <none>
I20260924 20:12:31.932930 22795 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:31.933027 22795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:31.933252 22795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:31.935396 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data/instance:
uuid: "b765fb07636e4bf297d01205650a61c5"
format_stamp: "Formatted at 2026-09-24 20:12:31 on dist-test-slave-2pm7"
server_key: "a441c19e8630920b1ea093cd9fc15fbb"
server_key_iv: "3654e15f749720f5ecee1e7cd346e87c"
server_key_version: "encryptionkey@0"
I20260924 20:12:31.936034 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/wal/instance:
uuid: "b765fb07636e4bf297d01205650a61c5"
format_stamp: "Formatted at 2026-09-24 20:12:31 on dist-test-slave-2pm7"
server_key: "a441c19e8630920b1ea093cd9fc15fbb"
server_key_iv: "3654e15f749720f5ecee1e7cd346e87c"
server_key_version: "encryptionkey@0"
I20260924 20:12:31.939821 22795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:12:31.942206 22810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:31.943423 22795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:31.943620 22795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/wal
uuid: "b765fb07636e4bf297d01205650a61c5"
format_stamp: "Formatted at 2026-09-24 20:12:31 on dist-test-slave-2pm7"
server_key: "a441c19e8630920b1ea093cd9fc15fbb"
server_key_iv: "3654e15f749720f5ecee1e7cd346e87c"
server_key_version: "encryptionkey@0"
I20260924 20:12:31.943738 22795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:31.976894 22795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:31.979964 22795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:31.980173 22795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:31.987270 22795 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.189:32931
I20260924 20:12:31.987555 22862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.189:32931 every 8 connection(s)
I20260924 20:12:31.988291 22795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
I20260924 20:12:31.991073 22863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:31.993731 22788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } attempt: 3
I20260924 20:12:31.994534 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22795
I20260924 20:12:31.994644 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-1/wal/instance
I20260924 20:12:31.994938 462 external_mini_cluster.cc:1428] Setting key 8e6bebb4ac1ab821348ab9e7b5eb7591
I20260924 20:12:31.995033 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:12:31 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
Sep 24 20:12:31 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.115.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.188@KRBTEST.COM" created.
I20260924 20:12:32.008461 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.994220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56838)
0924 20:12:31.994531 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.994550 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:31.994652 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:31.997297 (+ 2645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:31.997301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:31.997498 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:31.997668 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:31.997676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.000081 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.000085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.000750 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.000759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.001449 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.001456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.001601 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.002175 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.002190 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.003919 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.006242 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.006245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.006246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.006509 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.006897 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.006901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.006903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.007008 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.007730 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.007736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.007867 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.008021 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.008182 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260924 20:12:32.009416 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.996970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.0.115.190:42687)
0924 20:12:31.997102 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:31.997107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:31.997547 (+ 440us) server_negotiation.cc:407] Connection header received
0924 20:12:31.997744 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:31.997750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:31.997801 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:31.997878 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:31.998689 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:31.999217 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.000534 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.001006 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.001148 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.003781 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.003785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.003789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.003815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.006252 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.006821 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.006827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.006832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.006918 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.007324 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.007328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.007329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.007521 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.008753 (+ 1232us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.009172 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.009289 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:12:32.009793 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.995119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.115.189:32931)
0924 20:12:31.995590 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:31.995610 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:31.995653 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:12:31.996516 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:31.996542 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:31.996949 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:31.997847 (+ 898us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:31.998287 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:31.999935 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.000863 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.001314 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.001424 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.004093 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.004101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.004114 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.004141 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.006135 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.006630 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.006635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.006636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.006740 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.007103 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.007106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.007107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.007603 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.007734 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.008173 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.008449 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":320,"spinlock_wait_cycles":1408,"thread_start_us":150,"threads_started":1}
I20260924 20:12:32.009953 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:31.996916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:42687 (local address 127.0.0.1:58990)
0924 20:12:31.997405 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:31.997458 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:31.997642 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:31.997941 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:31.997945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:31.998284 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:31.998541 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:31.998555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:31.999324 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:31.999329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.000398 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.000409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.001160 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.001163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.001350 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.002001 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.002027 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.003576 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.006344 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.006355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.006365 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.006719 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.007045 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.007048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.007050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.007209 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.008807 (+ 1598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.008813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.008926 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.009596 (+ 670us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.009764 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":28,"spinlock_wait_cycles":3968,"thread_start_us":102,"threads_started":1}
I20260924 20:12:32.011075 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:32.011929 22788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:32.012687 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.012372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53632)
0924 20:12:32.012558 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.012614 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 20:12:32.013520 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.013174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53640)
0924 20:12:32.013384 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.013442 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260924 20:12:32.013682 22788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:46469: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
I20260924 20:12:32.019043 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.011598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56840)
0924 20:12:32.011728 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.011742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:32.011813 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:32.012158 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:32.012161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:32.012319 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:32.012387 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.012392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.013231 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.013237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.013902 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.013911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.014404 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.014407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.014504 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.014978 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.014988 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.016172 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.017586 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.017592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.017594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.017821 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.018094 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.018096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.018098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.018180 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.018517 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.018520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.018596 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.018703 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.018858 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 20:12:32.019053 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.011627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.0.115.189:32931)
0924 20:12:32.011978 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.011982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.011992 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:12:32.012020 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.012026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.012057 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.012145 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.012488 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.013133 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.014008 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.014303 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.014408 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.016298 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.016302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.016304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.016325 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.017485 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.017914 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.017918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.017920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.017991 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.018269 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.018272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.018274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.018431 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.018526 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.018756 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.018914 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":265}
I20260924 20:12:32.019982 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:32.020613 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.020388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53642)
0924 20:12:32.020491 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.020560 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 20:12:32.020838 22814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
I20260924 20:12:32.021695 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.021516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53658)
0924 20:12:32.021603 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.021648 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 20:12:32.021837 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:46469: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab.
Entry for principal kudu/127.0.115.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab.
I20260924 20:12:32.024830 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:46469
--webserver_interface=127.0.115.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab
--principal=kudu/127.0.115.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:42687,127.0.115.189:32931,127.0.115.188:46469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:12:32.044844 22788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } attempt: 1
I20260924 20:12:32.045759 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.045409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53668)
0924 20:12:32.045595 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.045667 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 20:12:32.046481 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.046269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53684)
0924 20:12:32.046373 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.046423 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 20:12:32.046615 22788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:46469: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
I20260924 20:12:32.071128 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } attempt: 1
I20260924 20:12:32.072113 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.071757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53698)
0924 20:12:32.071920 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.072026 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 20:12:32.072918 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.072684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53708)
0924 20:12:32.072798 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.072855 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 20:12:32.073086 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:46469: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
I20260924 20:12:32.121775 22788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } attempt: 2
I20260924 20:12:32.122637 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.122286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53720)
0924 20:12:32.122452 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.122529 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 20:12:32.123365 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.123140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53730)
0924 20:12:32.123252 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.123304 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 20:12:32.123497 22788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:46469: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
W20260924 20:12:32.137938 22870 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:12:32.138211 22870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:32.138234 22870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:32.138271 22870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:12:32.141254 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } attempt: 2
I20260924 20:12:32.142035 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.141760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53742)
0924 20:12:32.141898 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.141962 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:12:32.142479 22870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:32.142545 22870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:32.142585 22870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:32.142634 22870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:32.142668 22870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:32.142800 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.142548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53752)
0924 20:12:32.142688 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.142737 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 20:12:32.142918 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:46469: Network error: Client connection negotiation failed: client connection to 127.0.115.188:46469: connect: Connection refused (error 111)
I20260924 20:12:32.148087 22870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:42687,127.0.115.189:32931,127.0.115.188:46469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab
--principal=kudu/127.0.115.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:46469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.115.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:32.149168 22870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:32.150251 22870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:32.158754 22878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:32.158754 22875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:32.158880 22870 server_base.cc:1034] running on GCE node
W20260924 20:12:32.158792 22876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:32.159724 22870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:32.160902 22870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:32.162073 22870 hybrid_clock.cc:648] HybridClock initialized: now 1790280752162038 us; error 46 us; skew 500 ppm
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:32.164801 22870 init.cc:377] Logged in from keytab as kudu/127.0.115.188@KRBTEST.COM (short username kudu)
I20260924 20:12:32.165602 22870 webserver.cc:460] Webserver started at http://127.0.115.188:43625/ using document root <none> and password file <none>
I20260924 20:12:32.166118 22870 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:32.166174 22870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:32.166333 22870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:32.168376 22870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data/instance:
uuid: "53c7b0a6be3b4960a896c536c9f70139"
format_stamp: "Formatted at 2026-09-24 20:12:32 on dist-test-slave-2pm7"
server_key: "796e4291a54f4849980589265ee354a5"
server_key_iv: "98c895273c2eac41b5cc7eaf3b47853f"
server_key_version: "encryptionkey@0"
I20260924 20:12:32.168888 22870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/wal/instance:
uuid: "53c7b0a6be3b4960a896c536c9f70139"
format_stamp: "Formatted at 2026-09-24 20:12:32 on dist-test-slave-2pm7"
server_key: "796e4291a54f4849980589265ee354a5"
server_key_iv: "98c895273c2eac41b5cc7eaf3b47853f"
server_key_version: "encryptionkey@0"
I20260924 20:12:32.172397 22870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:12:32.174726 22885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:32.176110 22870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:32.176204 22870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/wal
uuid: "53c7b0a6be3b4960a896c536c9f70139"
format_stamp: "Formatted at 2026-09-24 20:12:32 on dist-test-slave-2pm7"
server_key: "796e4291a54f4849980589265ee354a5"
server_key_iv: "98c895273c2eac41b5cc7eaf3b47853f"
server_key_version: "encryptionkey@0"
I20260924 20:12:32.176281 22870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:32.186479 22870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:32.189222 22870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:32.189369 22870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:32.196022 22870 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.188:46469
I20260924 20:12:32.196309 22937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.188:46469 every 8 connection(s)
I20260924 20:12:32.196996 22870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
I20260924 20:12:32.199718 22938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:32.200905 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 22870
I20260924 20:12:32.201033 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790280466534371-462-0/minicluster-data/master-2/wal/instance
I20260924 20:12:32.201413 462 external_mini_cluster.cc:1428] Setting key 534468bb8f656263b22fa30c74c97e8f
I20260924 20:12:32.204348 22938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:32.214921 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.206039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.0.115.190:42687)
0924 20:12:32.206172 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.206178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.206448 (+ 270us) server_negotiation.cc:407] Connection header received
0924 20:12:32.206664 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.206667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.206712 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.206795 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.207522 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.208071 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.208949 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.209310 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.209421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.211674 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.211677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.211679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.211703 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.213004 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.213477 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.213479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.213481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.213568 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.213898 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.213901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.213902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.214080 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.214201 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.214582 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.214734 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:12:32.215971 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.202809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.0.115.188:46469)
0924 20:12:32.203262 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.203277 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.203313 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:12:32.204002 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.204028 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.204364 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.204681 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.205484 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.206929 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.207841 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.208385 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.208553 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.211164 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.211173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.211189 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.211212 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.212972 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.213391 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.213400 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.213405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.213549 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.213831 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.213837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.213841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.214280 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.214491 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.214664 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.214932 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":315,"thread_start_us":146,"threads_started":1}
I20260924 20:12:32.215971 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.205974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:42687 (local address 127.0.0.1:58992)
0924 20:12:32.206329 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.206366 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:32.206505 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:32.206846 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:32.206852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:32.207201 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:32.207386 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.207398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.208181 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.208185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.208830 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.208842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.209467 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.209470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.209554 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.210039 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.210062 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.211526 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.213128 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.213141 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.213149 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.213375 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.213684 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.213686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.213687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.213807 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.214177 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.214183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.214341 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.214530 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.215093 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":118,"spinlock_wait_cycles":10496,"thread_start_us":118,"threads_started":1}
I20260924 20:12:32.217262 22938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
I20260924 20:12:32.222653 22788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } attempt: 3
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:32.226375 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.217964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.0.115.189:32931)
0924 20:12:32.218090 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.218093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.218105 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:12:32.218150 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.218152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.218188 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.218271 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.218731 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.219300 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.220398 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.220645 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.220755 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.222946 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.222950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.222952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.222978 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.224613 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.225071 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.225074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.225076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.225165 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.225435 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.225439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.225441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.225622 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.225727 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.226076 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.226201 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:12:32.226828 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.217810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:32931 (local address 127.0.0.1:56856)
0924 20:12:32.217964 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.217978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:32.218050 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:32.218298 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:32.218300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:32.218439 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:32.218534 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.218543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.219449 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.219452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.220281 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.220290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.220755 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.220758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.220841 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.221279 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.221292 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.222812 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.224727 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.224730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.224732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.224950 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.225275 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.225277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.225279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.225344 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.225772 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.225777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.225874 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.226529 (+ 655us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.226682 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 20:12:32.227547 22938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:32.227857 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.218519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.0.115.190:42687)
0924 20:12:32.218638 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.218644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.218659 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:12:32.218732 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.218734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.218775 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.218847 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.219405 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.220175 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.220906 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.221142 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.221246 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.224045 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.224048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.224055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.224078 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.225591 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.226079 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.226083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.226085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.226166 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.226615 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.226619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.226620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.226832 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.227227 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.227568 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.227681 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
I20260924 20:12:32.231976 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.223014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53758)
0924 20:12:32.223305 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.223316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:32.223408 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:32.223792 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:32.223795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:32.224014 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:32.224143 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.224152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.225222 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.225225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.225936 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.225943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.226373 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.226377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.226462 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.226868 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.226889 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.228692 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.230457 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.230460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.230462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.230700 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.230993 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.230996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.230998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.231100 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.231491 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.231494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.231592 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.231666 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.231792 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222}
I20260924 20:12:32.232385 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.223189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.0.115.188:46469)
0924 20:12:32.223566 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.223572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.223586 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:12:32.223620 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.223622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.223664 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.224265 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.224354 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.225107 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.226040 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.226268 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.226370 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.228806 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.228810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.228812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.228836 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.230350 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.230817 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.230820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.230822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.230910 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.231181 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.231184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.231185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.231377 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.232077 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.232158 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.232269 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
I20260924 20:12:32.234053 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } attempt: 3
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
I20260924 20:12:32.236120 22788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2: Bootstrap starting.
I20260924 20:12:32.237066 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.227980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53768)
0924 20:12:32.228080 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.228093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:32.228164 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:32.228581 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:32.228584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:32.228753 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:32.228852 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.228859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.230258 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.230261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.231025 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.231034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.231639 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.231641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.231689 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.232276 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.232289 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.233649 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.235608 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.235610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.235612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.235852 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.236135 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.236138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.236140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.236211 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.236612 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.236615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.236714 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.236808 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.236921 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 20:12:32.237496 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.228071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.0.115.188:46469)
0924 20:12:32.228387 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.228395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.228406 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:12:32.228440 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.228446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.228490 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.228566 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.228966 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.230152 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.231138 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.231463 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.231560 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.233776 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.233779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.233781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.233799 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.235507 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.235967 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.235970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.235972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.236046 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.236297 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.236300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.236301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.236505 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.236584 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.237235 (+ 651us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.237333 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
I20260924 20:12:32.239197 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.228969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.0.115.189:32931)
0924 20:12:32.229356 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.229362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.229377 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:12:32.229416 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.229419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.229461 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.229809 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.230257 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.230836 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.232117 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.232474 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.232579 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.235562 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.235566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.235568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.235594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.237267 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.237933 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.237936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.237938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.238022 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.238337 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.238342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.238343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.238558 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.238778 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.238937 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.239044 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:12:32.239390 22788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:32.240235 22788 log.cc:824] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:32.240298 22938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: Bootstrap starting.
I20260924 20:12:32.242405 22788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2: No bootstrap required, opened a new log
Sep 24 20:12:32 dist-test-slave-2pm7 krb5kdc[22707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
I20260924 20:12:32.242713 22938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:32.243472 22938 log.cc:824] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:32.246263 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.234415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:46469 (local address 127.0.0.1:53772)
0924 20:12:32.234719 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:32.234733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:32.234811 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:32.235376 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:32.235380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:32.235551 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:32.235648 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.235656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.236745 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.236749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.237518 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:32.237531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.240875 (+ 3344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.240879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:32.240958 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.241358 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:32.241372 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:32.242784 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:32.244729 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.244732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.244734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.244968 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.245277 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:32.245280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:32.245282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:32.245365 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:32.245786 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:32.245790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:32.245878 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 20:12:32.245972 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 20:12:32.246100 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260924 20:12:32.246771 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.234968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.0.115.188:46469)
0924 20:12:32.235142 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.235147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.235159 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:12:32.235194 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.235197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.235239 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.235758 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.235851 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.236631 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.237996 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.240746 (+ 2750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.240869 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.242944 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.242948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.242950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.242973 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.244608 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.245090 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.245093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.245095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.245174 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.245470 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.245473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.245474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.245675 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.246402 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.246478 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.246638 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 20:12:32.246773 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:32.246765 22788 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.247012 22788 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e8e4ebe30b47aea03a8850317ff8f2, State: Initialized, Role: FOLLOWER
I20260924 20:12:32.247319 22788 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.247820 22946 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.248009 22946 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:32.248380 22762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59006
I20260924 20:12:32.248690 22938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: No bootstrap required, opened a new log
I20260924 20:12:32.248697 22788 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:32.248775 22762 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 20:12:32.248907 22762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:32.250135 22762 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.250689 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: Bootstrap starting.
I20260924 20:12:32.251063 22762 leader_election.cc:288] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [CANDIDATE]: Term 1 election: Requested vote from peers b765fb07636e4bf297d01205650a61c5 (127.0.115.189:32931), 53c7b0a6be3b4960a896c536c9f70139 (127.0.115.188:46469)
I20260924 20:12:32.253185 22938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.253499 22938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c7b0a6be3b4960a896c536c9f70139, State: Initialized, Role: FOLLOWER
I20260924 20:12:32.253873 22938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.254431 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.254686 22863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:32.254675 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:32.255362 22938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:32.255595 22863 log.cc:824] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:32.257220 22912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "53c7b0a6be3b4960a896c536c9f70139"
I20260924 20:12:32.257452 22912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:32.258018 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: No bootstrap required, opened a new log
I20260924 20:12:32.258801 22912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e8e4ebe30b47aea03a8850317ff8f2 in term 1.
I20260924 20:12:32.259218 22739 leader_election.cc:302] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c7b0a6be3b4960a896c536c9f70139, b0e8e4ebe30b47aea03a8850317ff8f2; no voters:
I20260924 20:12:32.253988 22720 master_runner.cc:186] Error getting master registration for 127.0.115.189:32931: OK, instance_id { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" instance_seqno: 1790280751985357 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:12:32.260195 22720 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:12:32.261023 22946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:32.261231 22946 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [term 1 LEADER]: Becoming Leader. State: Replica: b0e8e4ebe30b47aea03a8850317ff8f2, State: Running, Role: LEADER
I20260924 20:12:32.261567 22946 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.262106 22863 raft_consensus.cc:348] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.262305 22863 raft_consensus.cc:729] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b765fb07636e4bf297d01205650a61c5, State: Initialized, Role: FOLLOWER
I20260924 20:12:32.263007 22863 consensus_queue.cc:260] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } }
I20260924 20:12:32.263434 22946 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e8e4ebe30b47aea03a8850317ff8f2. Latest consensus state: current_term: 1 leader_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.263567 22946 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: This master's current role is: LEADER
W20260924 20:12:32.264627 22957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:32.264698 22957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:32.265642 22837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b765fb07636e4bf297d01205650a61c5"
I20260924 20:12:32.265877 22837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:32.267223 22837 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e8e4ebe30b47aea03a8850317ff8f2 in term 1.
W20260924 20:12:32.269156 22978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:32.269228 22978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:32.269546 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.256307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.0.115.190:42687)
0924 20:12:32.256419 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.256423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.257964 (+ 1541us) server_negotiation.cc:407] Connection header received
0924 20:12:32.258164 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.258168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.258213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.258263 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.259278 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.260082 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.261285 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.261637 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.261712 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.265911 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.265916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.265919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.265946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.267643 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.268215 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.268219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.268221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.268316 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.268613 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.268616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.268618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.268830 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.268925 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.269277 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.269393 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:12:32.268229 22870 master_runner.cc:186] Error getting master registration for 127.0.115.189:32931: OK, instance_id { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" instance_seqno: 1790280751985357 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:12:32.271399 22870 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:12:32.265574 22863 sys_catalog.cc:564] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:32.276444 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.257405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.115.188:46469)
0924 20:12:32.257510 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.257514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.266953 (+ 9439us) server_negotiation.cc:407] Connection header received
0924 20:12:32.267146 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.267149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.267206 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.267256 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.268361 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.269080 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.270108 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.270396 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.270474 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.272075 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.272079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.272082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.272108 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.274238 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.274745 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.274750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.274752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.274840 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.275155 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.275159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.275160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.275386 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.275502 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.276169 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.276285 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:12:32.277770 22982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:32.277884 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.278075 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:32.278407 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:32.257041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.0.115.189:32931)
0924 20:12:32.257151 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.257155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.261418 (+ 4263us) server_negotiation.cc:407] Connection header received
0924 20:12:32.261597 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.261601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.261657 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.261710 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.262671 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.263354 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.264496 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:32.264832 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:32.264912 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:32.272075 (+ 7163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:32.272079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:32.272082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:32.272108 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:32.273987 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.277063 (+ 3076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.277066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.277070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.277160 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:32.277472 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:32.277475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:32.277477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:32.277696 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:32.277803 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:32.278157 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 20:12:32.278274 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:12:32.278816 22982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:12:32.284060 22912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [term 1 FOLLOWER]: Refusing update from remote peer b0e8e4ebe30b47aea03a8850317ff8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:32.284843 22979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:32.289134 22837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [term 1 FOLLOWER]: Refusing update from remote peer b0e8e4ebe30b47aea03a8850317ff8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:32.290360 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e8e4ebe30b47aea03a8850317ff8f2. Latest consensus state: current_term: 1 leader_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.290501 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:32.290684 22979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:32.294132 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e8e4ebe30b47aea03a8850317ff8f2. Latest consensus state: current_term: 1 leader_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.294250 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 20:12:32.296662 22994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:32.296744 22994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:32.300882 22946 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.301012 22946 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:32.301401 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.301506 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:32.304052 22999 mvcc.cc:205] Tried to move back new op lower bound from 7332989961347780608 to 7332989961267200000. Current Snapshot: MvccSnapshot[applied={T|T < 7332989961347780608}]
I20260924 20:12:32.308015 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.308149 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:32.310806 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e8e4ebe30b47aea03a8850317ff8f2" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 42687 } } peers { permanent_uuid: "b765fb07636e4bf297d01205650a61c5" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 32931 } } peers { permanent_uuid: "53c7b0a6be3b4960a896c536c9f70139" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 46469 } } }
I20260924 20:12:32.310932 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:32.313704 22997 mvcc.cc:205] Tried to move back new op lower bound from 7332989961347780608 to 7332989961267200000. Current Snapshot: MvccSnapshot[applied={T|T < 7332989961347780608}]
I20260924 20:12:32.313854 22982 catalog_manager.cc:1347] Generated new cluster ID: b73ef26ed5074cde9f4f76f384ba0d22
I20260924 20:12:32.313920 22982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:32.342135 22982 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:32.343520 22982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:32.368412 22982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e8e4ebe30b47aea03a8850317ff8f2: Generated new TSK 0
I20260924 20:12:32.369387 22982 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:12:32.401767 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.398736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.0.115.189:32931)
0924 20:12:32.398871 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.398876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.400377 (+ 1501us) server_negotiation.cc:407] Connection header received
0924 20:12:32.400818 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.400822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.400876 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.400949 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.401615 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56868: BlockingRecv error: recv got EOF from 127.0.0.1:56868 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 20:12:32.402886 22941 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.399479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.115.188:46469)
0924 20:12:32.399605 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.399609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.400729 (+ 1120us) server_negotiation.cc:407] Connection header received
0924 20:12:32.402118 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.402121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.402171 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.402222 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.402768 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53780: BlockingRecv error: recv got EOF from 127.0.0.1:53780 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 20:12:32.405037 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.403728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.115.190:42687)
0924 20:12:32.403843 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.403848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.404158 (+ 310us) server_negotiation.cc:407] Connection header received
0924 20:12:32.404323 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.404326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.404376 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.404426 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.404915 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59032: BlockingRecv error: recv got EOF from 127.0.0.1:59032 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 20:12:32.409531 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.407898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.0.115.189:32931)
0924 20:12:32.408052 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.408056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.408573 (+ 517us) server_negotiation.cc:407] Connection header received
0924 20:12:32.408791 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.408794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.408870 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.408921 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.409413 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56878: BlockingRecv error: recv got EOF from 127.0.0.1:56878 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 20:12:32.414330 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.412836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.115.189:32931)
0924 20:12:32.412930 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.412934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.413423 (+ 489us) server_negotiation.cc:407] Connection header received
0924 20:12:32.413630 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.413633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.413681 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.413732 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.414217 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56886: BlockingRecv error: recv got EOF from 127.0.0.1:56886 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 20:12:32.419652 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.418231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.115.189:32931)
0924 20:12:32.418324 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.418328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.418819 (+ 491us) server_negotiation.cc:407] Connection header received
0924 20:12:32.418989 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.418993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.419041 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.419089 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.419552 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56890: BlockingRecv error: recv got EOF from 127.0.0.1:56890 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 20:12:32.427245 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.425648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.0.115.189:32931)
0924 20:12:32.425780 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.425784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.426309 (+ 525us) server_negotiation.cc:407] Connection header received
0924 20:12:32.426498 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.426514 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.426565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.426618 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.427136 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56898: BlockingRecv error: recv got EOF from 127.0.0.1:56898 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 20:12:32.439373 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.437633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.0.115.189:32931)
0924 20:12:32.437761 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.437767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.438353 (+ 586us) server_negotiation.cc:407] Connection header received
0924 20:12:32.438578 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.438582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.438640 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.438695 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.439252 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56900: BlockingRecv error: recv got EOF from 127.0.0.1:56900 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:12:32.460016 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.458047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.0.115.189:32931)
0924 20:12:32.458180 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.458184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.458887 (+ 703us) server_negotiation.cc:407] Connection header received
0924 20:12:32.459122 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.459126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.459185 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.459242 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.459852 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56908: BlockingRecv error: recv got EOF from 127.0.0.1:56908 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:12:32.495913 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.494642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.0.115.189:32931)
0924 20:12:32.494820 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.494824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.494948 (+ 124us) server_negotiation.cc:407] Connection header received
0924 20:12:32.495071 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.495074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.495122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.495195 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.495813 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56916: BlockingRecv error: recv got EOF from 127.0.0.1:56916 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 20:12:32.563220 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.561982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.0.115.189:32931)
0924 20:12:32.562163 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.562168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.562303 (+ 135us) server_negotiation.cc:407] Connection header received
0924 20:12:32.562486 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.562489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.562538 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.562657 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.563110 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56924: BlockingRecv error: recv got EOF from 127.0.0.1:56924 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 20:12:32.694437 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.693352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.115.189:32931)
0924 20:12:32.693531 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.693535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.693665 (+ 130us) server_negotiation.cc:407] Connection header received
0924 20:12:32.693794 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.693797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.693858 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.693942 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.694335 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56936: BlockingRecv error: recv got EOF from 127.0.0.1:56936 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 20:12:32.953792 22865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:32.952603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.0.115.189:32931)
0924 20:12:32.952799 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:32.952803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:32.952888 (+ 85us) server_negotiation.cc:407] Connection header received
0924 20:12:32.953021 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:32.953023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:32.953074 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:32.953154 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:32.953652 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56950: BlockingRecv error: recv got EOF from 127.0.0.1:56950 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:12:33.270619 22978 catalog_manager.cc:1259] Loaded cluster ID: b73ef26ed5074cde9f4f76f384ba0d22
I20260924 20:12:33.270741 22978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: loading cluster ID for follower catalog manager: success
I20260924 20:12:33.272648 22978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: acquiring CA information for follower catalog manager: success
I20260924 20:12:33.273959 22978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53c7b0a6be3b4960a896c536c9f70139: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:12:33.298075 22994 catalog_manager.cc:1259] Loaded cluster ID: b73ef26ed5074cde9f4f76f384ba0d22
I20260924 20:12:33.298187 22994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: loading cluster ID for follower catalog manager: success
I20260924 20:12:33.300096 22994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: acquiring CA information for follower catalog manager: success
I20260924 20:12:33.301316 22994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b765fb07636e4bf297d01205650a61c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:12:33.472873 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:33.468198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.0.115.189:32931)
0924 20:12:33.468490 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:33.468495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:33.468615 (+ 120us) server_negotiation.cc:407] Connection header received
0924 20:12:33.468803 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:33.468806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:33.468934 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:33.469085 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:33.469356 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.469912 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.470818 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.471068 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.471180 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:33.471622 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:33.472191 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:33.472301 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:33.472581 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 20:12:33.472682 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260924 20:12:33.480615 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:33.476559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.0.115.188:46469)
0924 20:12:33.476804 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:33.476808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:33.476822 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:12:33.476909 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:33.476912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:33.477036 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:33.477144 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:33.477440 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.477971 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.478864 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.479095 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.479185 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:33.479587 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:33.480173 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:33.480274 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:33.480363 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 20:12:33.480416 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260924 20:12:33.489857 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:33.484695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.115.188:46469)
0924 20:12:33.484820 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:33.484825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:33.485763 (+ 938us) server_negotiation.cc:407] Connection header received
0924 20:12:33.485963 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:33.485966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:33.486014 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:33.486059 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:33.486533 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.487265 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.488581 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.488921 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.488990 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:33.489368 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:33.489447 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:33.489493 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:33.489667 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 20:12:33.489724 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:12:33.489871 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:33.483609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.115.190:42687)
0924 20:12:33.484551 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:33.484556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:33.484870 (+ 314us) server_negotiation.cc:407] Connection header received
0924 20:12:33.484971 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:33.484977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:33.485166 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:33.485283 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:33.485621 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.486405 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.487699 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.488101 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.488523 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:33.488712 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:33.489103 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:33.489602 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:33.489641 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 20:12:33.489710 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":87,"threads_started":1}
I20260924 20:12:33.489954 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:33.483680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.0.115.189:32931)
0924 20:12:33.483822 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:33.483827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:33.484335 (+ 508us) server_negotiation.cc:407] Connection header received
0924 20:12:33.484557 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:33.484572 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:33.484629 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:33.484679 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:33.485058 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.486506 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.488113 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:33.488604 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:33.488715 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:33.489238 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:33.489348 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:33.489422 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:33.489707 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 20:12:33.489790 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:12:33.493767 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22720
I20260924 20:12:33.500752 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22795
I20260924 20:12:33.507810 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 22870
2026-09-24T20:12:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3122 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:12:33 dist-test-slave-2pm7 krb5kdc[23085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:12:33 dist-test-slave-2pm7 krb5kdc[23085](info): set up 2 sockets
Sep 24 20:12:33 dist-test-slave-2pm7 krb5kdc[23085](info): commencing operation
krb5kdc: starting...
W20260924 20:12:34.548520 462 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/test-user.keytab.
Sep 24 20:12:34 dist-test-slave-2pm7 krb5kdc[23085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.115.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
Entry for principal kudu/127.0.115.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab.
I20260924 20:12:34.667131 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33171
--webserver_interface=127.0.115.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:33171,127.0.115.189:37581,127.0.115.188:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:12:34.778477 23098 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:12:34.778836 23098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:34.778905 23098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:34.778987 23098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:34.782955 23098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:34.783049 23098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:34.783094 23098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:34.783128 23098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:34.783160 23098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:34.788383 23098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:33171,127.0.115.189:37581,127.0.115.188:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.190.keytab
--principal=kudu/127.0.115.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.190:33171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.115.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:34.789466 23098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:34.790514 23098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:34.799098 23104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:34.799163 23106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:34.799175 23103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:34.799491 23098 server_base.cc:1034] running on GCE node
I20260924 20:12:34.800076 23098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:34.801191 23098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:34.802376 23098 hybrid_clock.cc:648] HybridClock initialized: now 1790280754802359 us; error 26 us; skew 500 ppm
Sep 24 20:12:34 dist-test-slave-2pm7 krb5kdc[23085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:34.805192 23098 init.cc:377] Logged in from keytab as kudu/127.0.115.190@KRBTEST.COM (short username kudu)
I20260924 20:12:34.806185 23098 webserver.cc:460] Webserver started at http://127.0.115.190:36481/ using document root <none> and password file <none>
I20260924 20:12:34.806751 23098 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:34.806836 23098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:34.807045 23098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:34.809166 23098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data/instance:
uuid: "3615bb6e4a114285bb5c27e58a731361"
format_stamp: "Formatted at 2026-09-24 20:12:34 on dist-test-slave-2pm7"
server_key: "35d5e67496d07587ba96e173b458e2c6"
server_key_iv: "6ecb15ba2ff80cd38b75f8def2422ff9"
server_key_version: "encryptionkey@0"
I20260924 20:12:34.809731 23098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/wal/instance:
uuid: "3615bb6e4a114285bb5c27e58a731361"
format_stamp: "Formatted at 2026-09-24 20:12:34 on dist-test-slave-2pm7"
server_key: "35d5e67496d07587ba96e173b458e2c6"
server_key_iv: "6ecb15ba2ff80cd38b75f8def2422ff9"
server_key_version: "encryptionkey@0"
I20260924 20:12:34.813792 23098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:12:34.816314 23113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:34.817499 23098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:12:34.817637 23098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/wal
uuid: "3615bb6e4a114285bb5c27e58a731361"
format_stamp: "Formatted at 2026-09-24 20:12:34 on dist-test-slave-2pm7"
server_key: "35d5e67496d07587ba96e173b458e2c6"
server_key_iv: "6ecb15ba2ff80cd38b75f8def2422ff9"
server_key_version: "encryptionkey@0"
I20260924 20:12:34.817747 23098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:34.848456 23098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:34.851799 23098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:34.852020 23098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:34.859041 23098 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.190:33171
I20260924 20:12:34.859331 23165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.190:33171 every 8 connection(s)
I20260924 20:12:34.860067 23098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/data/info.pb
I20260924 20:12:34.862816 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:34.863456 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 23098
I20260924 20:12:34.863581 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-0/wal/instance
I20260924 20:12:34.863910 462 external_mini_cluster.cc:1428] Setting key 1fffcc5ebcfa5fad90bccb599e72c8ec
I20260924 20:12:34.867429 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.115.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.189@KRBTEST.COM" created.
Sep 24 20:12:34 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:34.883030 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:34.869828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33171 (local address 127.0.0.1:33702)
0924 20:12:34.870395 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:34.870430 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:34.871232 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:34.871893 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:34.871898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:34.872340 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:34.872577 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:34.872590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:34.873866 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:34.873868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:34.874733 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:34.874744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:34.875549 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:34.875552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:34.875715 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:34.876323 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:34.876350 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:34.877942 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:34.880278 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:34.880285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:34.880293 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:34.880508 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:34.880879 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:34.880881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:34.880883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:34.880984 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:34.881730 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:34.881734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:34.881851 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 20:12:34.881989 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 20:12:34.882250 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":30,"spinlock_wait_cycles":33536,"thread_start_us":152,"threads_started":1}
I20260924 20:12:34.883036 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:34.869827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.0.115.190:33171)
0924 20:12:34.870260 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:34.870278 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:34.870531 (+ 253us) server_negotiation.cc:407] Connection header received
0924 20:12:34.871526 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:34.871530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:34.871713 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:34.871895 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:34.872726 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:34.873730 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:34.874844 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:34.875359 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:34.875545 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:34.878100 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:34.878110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:34.878126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:34.878159 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:34.880168 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:34.880619 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:34.880628 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:34.880631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:34.880781 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:34.881079 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:34.881086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:34.881089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:34.881616 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:34.881771 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:34.882109 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 20:12:34.882323 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260924 20:12:34.885522 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:34.886263 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:34.885987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48484)
0924 20:12:34.886140 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:34.886208 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 20:12:34.886502 23114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
I20260924 20:12:34.887349 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:34.887116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48488)
0924 20:12:34.887232 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:34.887281 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:12:34.887497 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:37581: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab.
Entry for principal kudu/127.0.115.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab.
I20260924 20:12:34.891577 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:37581
--webserver_interface=127.0.115.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab
--principal=kudu/127.0.115.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:33171,127.0.115.189:37581,127.0.115.188:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:12:34.936726 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } attempt: 1
I20260924 20:12:34.937582 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:34.937291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48494)
0924 20:12:34.937436 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:34.937505 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 20:12:34.938391 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:34.938180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48510)
0924 20:12:34.938286 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:34.938338 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:12:34.938555 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:37581: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
W20260924 20:12:35.002677 23173 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:12:35.002962 23173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:35.002986 23173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:35.003085 23173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:12:35.006716 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } attempt: 2
W20260924 20:12:35.007189 23173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:35.007264 23173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:35.007301 23173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:35.007339 23173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:35.007372 23173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:35.007529 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.007222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48526)
0924 20:12:35.007374 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.007447 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 20:12:35.008299 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.008083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48534)
0924 20:12:35.008190 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.008237 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 20:12:35.008451 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.189:37581: Network error: Client connection negotiation failed: client connection to 127.0.115.189:37581: connect: Connection refused (error 111)
I20260924 20:12:35.012858 23173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:33171,127.0.115.189:37581,127.0.115.188:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.189.keytab
--principal=kudu/127.0.115.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.189:37581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.115.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:35.013921 23173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:35.014947 23173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:35.023309 23179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:35.023320 23181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:35.023480 23173 server_base.cc:1034] running on GCE node
W20260924 20:12:35.023329 23178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:35.024186 23173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:35.025255 23173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:35.026445 23173 hybrid_clock.cc:648] HybridClock initialized: now 1790280755026423 us; error 26 us; skew 500 ppm
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:35.029170 23173 init.cc:377] Logged in from keytab as kudu/127.0.115.189@KRBTEST.COM (short username kudu)
I20260924 20:12:35.030055 23173 webserver.cc:460] Webserver started at http://127.0.115.189:39579/ using document root <none> and password file <none>
I20260924 20:12:35.030635 23173 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:35.030725 23173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:35.030936 23173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:35.032971 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data/instance:
uuid: "a3da11841d624f01a1bd51a9b1300d1f"
format_stamp: "Formatted at 2026-09-24 20:12:35 on dist-test-slave-2pm7"
server_key: "e4ef5cd93ddcd2a9aaf545d3da2fadd3"
server_key_iv: "d1252fff6d794ab49fbc7c4d2c745a41"
server_key_version: "encryptionkey@0"
I20260924 20:12:35.033524 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/wal/instance:
uuid: "a3da11841d624f01a1bd51a9b1300d1f"
format_stamp: "Formatted at 2026-09-24 20:12:35 on dist-test-slave-2pm7"
server_key: "e4ef5cd93ddcd2a9aaf545d3da2fadd3"
server_key_iv: "d1252fff6d794ab49fbc7c4d2c745a41"
server_key_version: "encryptionkey@0"
I20260924 20:12:35.037107 23173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 20:12:35.039383 23188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:35.040596 23173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:12:35.040745 23173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/wal
uuid: "a3da11841d624f01a1bd51a9b1300d1f"
format_stamp: "Formatted at 2026-09-24 20:12:35 on dist-test-slave-2pm7"
server_key: "e4ef5cd93ddcd2a9aaf545d3da2fadd3"
server_key_iv: "d1252fff6d794ab49fbc7c4d2c745a41"
server_key_version: "encryptionkey@0"
I20260924 20:12:35.040855 23173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:35.081184 23173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:35.084079 23173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:35.084285 23173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:35.091243 23173 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.189:37581
I20260924 20:12:35.091507 23240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.189:37581 every 8 connection(s)
I20260924 20:12:35.092263 23173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/data/info.pb
I20260924 20:12:35.095031 23241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:35.098079 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 23173
I20260924 20:12:35.098212 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-1/wal/instance
I20260924 20:12:35.098489 462 external_mini_cluster.cc:1428] Setting key cec576f317f6f88380df6ff9f00587f9
I20260924 20:12:35.098834 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:35.099619 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } attempt: 3
WARNING: no policy specified for kudu/127.0.115.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.115.188@KRBTEST.COM" created.
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
I20260924 20:12:35.116398 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.100144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48550)
0924 20:12:35.100353 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.100370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.100464 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.103147 (+ 2683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.103150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.103362 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.103487 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.103498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.105482 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.105486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.106348 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.106358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.107058 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.107072 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.107218 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.107723 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.107737 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.111873 (+ 4136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.114653 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.114656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.114658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.114898 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.115222 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.115227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.115230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.115313 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.115826 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.115829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.115926 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.116085 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.116225 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":81}
I20260924 20:12:35.116407 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.100654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.0.115.190:33171)
0924 20:12:35.100754 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.100758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.101469 (+ 711us) server_negotiation.cc:407] Connection header received
0924 20:12:35.102822 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.102825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.102885 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.102969 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.104056 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.104573 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.105902 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.106545 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.106675 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.111460 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.111468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.111470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.111508 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.113798 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.114376 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.114379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.114381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.114459 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.114887 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.114891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.114892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.115096 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.115156 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.115976 (+ 820us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.116129 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":41}
I20260924 20:12:35.117712 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.100875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.0.115.189:37581)
0924 20:12:35.101438 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.101451 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.101494 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:12:35.102598 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.102625 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.102899 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.103169 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.103623 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.105330 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.106456 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.106923 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.107032 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.112048 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.112055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.112077 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.112127 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.114484 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.114996 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.115006 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.115010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.115116 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.115412 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.115417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.115420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.115719 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.115859 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.116188 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.116486 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":34176,"thread_start_us":172,"threads_started":1}
I20260924 20:12:35.117728 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.100874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33171 (local address 127.0.0.1:33714)
0924 20:12:35.101329 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.101379 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.102604 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.103026 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.103035 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.103535 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.103896 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.103909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.104695 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.104698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.105765 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.105776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.106706 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.106710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.106881 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.107556 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.107581 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.111313 (+ 3732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.113911 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.113922 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.113932 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.114275 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.114566 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.114569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.114571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.114765 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.115259 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.115269 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.115624 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.116435 (+ 811us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.116673 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":185,"thread_start_us":132,"threads_started":1}
I20260924 20:12:35.119375 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:35.119634 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:35.120342 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.120106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59340)
0924 20:12:35.120216 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.120273 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 20:12:35.121107 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.120821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59356)
0924 20:12:35.120974 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.121024 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 20:12:35.121259 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:43657: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.115.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab.
Entry for principal kudu/127.0.115.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab.
I20260924 20:12:35.127532 462 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
/tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:43657
--webserver_interface=127.0.115.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab
--principal=kudu/127.0.115.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.115.148:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.115.190:33171,127.0.115.189:37581,127.0.115.188:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:12:35.128101 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.119861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48560)
0924 20:12:35.120052 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.120075 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.120148 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.120427 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.120431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.120605 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.120717 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.120728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.121673 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.121677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.122445 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.122459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.122980 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.122984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.123076 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.123462 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.123474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.124909 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.126659 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.126661 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.126662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.126825 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.127125 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.127127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.127129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.127198 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.127604 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.127606 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.127743 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.127843 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.127983 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 20:12:35.128340 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.120131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.0.115.189:37581)
0924 20:12:35.120221 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.120227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.120238 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:12:35.120273 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.120276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.120314 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.120391 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.120816 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.121554 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.122563 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.122849 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.122929 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.125047 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.125050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.125052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.125072 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.126522 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.126907 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.126911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.126913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.127005 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.127284 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.127290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.127295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.127514 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.127621 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.127908 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.128116 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:12:35.129096 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:12:35.129763 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.129465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59362)
0924 20:12:35.129592 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.129686 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 20:12:35.129982 23192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
I20260924 20:12:35.132570 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.132375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59364)
0924 20:12:35.132476 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.132522 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 20:12:35.132746 23241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:43657: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
I20260924 20:12:35.152421 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } attempt: 1
I20260924 20:12:35.153255 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.152960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59374)
0924 20:12:35.153108 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.153178 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 20:12:35.153964 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.153747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59386)
0924 20:12:35.153859 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.153906 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:12:35.154093 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:43657: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
I20260924 20:12:35.182042 23241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } attempt: 1
I20260924 20:12:35.182981 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.182682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59392)
0924 20:12:35.182840 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.182905 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 20:12:35.183688 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.183476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59394)
0924 20:12:35.183582 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.183630 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 20:12:35.183815 23241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:43657: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
I20260924 20:12:35.229266 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } attempt: 2
I20260924 20:12:35.230115 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.229820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59404)
0924 20:12:35.229973 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.230036 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 20:12:35.230908 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.230658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59406)
0924 20:12:35.230778 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.230835 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:12:35.231076 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:43657: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
W20260924 20:12:35.241750 23247 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:12:35.242057 23247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:12:35.242105 23247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:12:35.242190 23247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:12:35.246407 23247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:12:35.246476 23247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:12:35.246496 23247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:12:35.246546 23247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:12:35.246587 23247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:12:35.252012 23241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } attempt: 2
I20260924 20:12:35.252429 23247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.115.148:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.115.190:33171,127.0.115.189:37581,127.0.115.188:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/krb5kdc/kudu_127.0.115.188.keytab
--principal=kudu/127.0.115.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.115.188:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.115.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:12:35.252821 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.252531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59408)
0924 20:12:35.252679 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.252747 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 20:12:35.253551 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.253342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59412)
0924 20:12:35.253447 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.253491 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 20:12:35.253695 23241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.115.188:43657: Network error: Client connection negotiation failed: client connection to 127.0.115.188:43657: connect: Connection refused (error 111)
I20260924 20:12:35.253865 23247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:12:35.254914 23247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:12:35.263644 23253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:35.263634 23254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:12:35.263698 23256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:12:35.263839 23247 server_base.cc:1034] running on GCE node
I20260924 20:12:35.264458 23247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:12:35.265616 23247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:12:35.266801 23247 hybrid_clock.cc:648] HybridClock initialized: now 1790280755266784 us; error 27 us; skew 500 ppm
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:12:35.269434 23247 init.cc:377] Logged in from keytab as kudu/127.0.115.188@KRBTEST.COM (short username kudu)
I20260924 20:12:35.270310 23247 webserver.cc:460] Webserver started at http://127.0.115.188:39995/ using document root <none> and password file <none>
I20260924 20:12:35.270870 23247 fs_manager.cc:359] Metadata directory not provided
I20260924 20:12:35.270965 23247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:12:35.271193 23247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:12:35.273285 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data/instance:
uuid: "368936bfe9684a99a8d7a264865e15b0"
format_stamp: "Formatted at 2026-09-24 20:12:35 on dist-test-slave-2pm7"
server_key: "a33173fe01b22396b6f4a38bd49ac41f"
server_key_iv: "7d8054d20d8c0f4001478b8da7ee4a5b"
server_key_version: "encryptionkey@0"
I20260924 20:12:35.273857 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/wal/instance:
uuid: "368936bfe9684a99a8d7a264865e15b0"
format_stamp: "Formatted at 2026-09-24 20:12:35 on dist-test-slave-2pm7"
server_key: "a33173fe01b22396b6f4a38bd49ac41f"
server_key_iv: "7d8054d20d8c0f4001478b8da7ee4a5b"
server_key_version: "encryptionkey@0"
I20260924 20:12:35.277630 23247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:12:35.280082 23263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:12:35.281296 23247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:12:35.281441 23247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data,/tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/wal
uuid: "368936bfe9684a99a8d7a264865e15b0"
format_stamp: "Formatted at 2026-09-24 20:12:35 on dist-test-slave-2pm7"
server_key: "a33173fe01b22396b6f4a38bd49ac41f"
server_key_iv: "7d8054d20d8c0f4001478b8da7ee4a5b"
server_key_version: "encryptionkey@0"
I20260924 20:12:35.281538 23247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:12:35.291272 23247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:12:35.294088 23247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:12:35.294238 23247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:12:35.300994 23247 rpc_server.cc:307] RPC server started. Bound to: 127.0.115.188:43657
I20260924 20:12:35.301291 23315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.115.188:43657 every 8 connection(s)
I20260924 20:12:35.301975 23247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/data/info.pb
I20260924 20:12:35.304266 462 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu as pid 23247
I20260924 20:12:35.304425 462 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIXC3Dg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790280466534371-462-0/minicluster-data/master-2/wal/instance
I20260924 20:12:35.304745 462 external_mini_cluster.cc:1428] Setting key 891b59d42b9809bc9cde89a1feb0ee35
I20260924 20:12:35.305164 23316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:12:35.310073 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:35.320721 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.306211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.115.188:43657)
0924 20:12:35.306844 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.306868 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.306918 (+ 50us) server_negotiation.cc:407] Connection header received
0924 20:12:35.308007 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.308038 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.308468 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.308802 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.309639 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.310695 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.311561 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.312229 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.312396 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.315102 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.315107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.315117 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.315144 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.316926 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.317371 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.317378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.317385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.317528 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.318178 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.318183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.318185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.318775 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.318978 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.319296 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.319610 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":155,"threads_started":1}
I20260924 20:12:35.321410 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.311798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.190:33171 (local address 127.0.0.1:33724)
0924 20:12:35.312246 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.312297 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.312465 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.312927 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.312934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.313312 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.313439 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.313445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.314385 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.314389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.315096 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.315106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.315739 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.315743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.315866 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.316441 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.316452 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.317892 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.319745 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.319753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.319758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.320021 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.320304 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.320306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.320307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.320378 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.320807 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.320815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.320954 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.321096 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.321280 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":168,"threads_started":1}
I20260924 20:12:35.321439 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.311884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.0.115.190:33171)
0924 20:12:35.312053 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.312058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.312377 (+ 319us) server_negotiation.cc:407] Connection header received
0924 20:12:35.312576 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.312582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.312646 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.312724 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.313581 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.314256 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.315231 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.315569 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.315708 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.318018 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.318020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.318022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.318047 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.319645 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.320108 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.320112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.320114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.320201 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.320467 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.320470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.320472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.320694 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.320802 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.321171 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.321298 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 20:12:35.322496 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.190@KRBTEST.COM
I20260924 20:12:35.330264 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } attempt: 3
I20260924 20:12:35.331139 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.323114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.0.115.189:37581)
0924 20:12:35.323265 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.323270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.323285 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:12:35.323440 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.323443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.323499 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.323600 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.324015 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.324515 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.325281 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.325541 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.325646 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.327885 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.327889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.327892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.327920 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.329323 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.329792 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.329797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.329799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.329891 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.330158 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.330162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.330164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.330412 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.330519 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.330858 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.330988 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:12:35.331981 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.323011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.189:37581 (local address 127.0.0.1:48574)
0924 20:12:35.323215 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.323230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.323310 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.323615 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.323618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.323787 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.323876 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.323881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.324615 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.324621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.325150 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.325158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.325648 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.325651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.325743 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.326221 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.326237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.327740 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.329431 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.329437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.329442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.329690 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.329987 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.329988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.329990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.330059 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.330505 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.330508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.330635 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.331667 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.331812 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 20:12:35.332438 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.323408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.0.115.190:33171)
0924 20:12:35.323529 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.323533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.323640 (+ 107us) server_negotiation.cc:407] Connection header received
0924 20:12:35.323786 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.323789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.323828 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.323910 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.324524 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.325006 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.325753 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.325978 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.326082 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.328717 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.328724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.328730 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.328763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.330427 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.331159 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.331163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.331164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.331268 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.331522 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.331526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.331527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.331745 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.332014 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.332193 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.332304 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:12:35.332834 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.190@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.115.189@KRBTEST.COM
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.188@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
I20260924 20:12:35.340083 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.330693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59424)
0924 20:12:35.330819 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.330833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.330902 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.331612 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.331614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.331756 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.331873 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.331879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.333319 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.333322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.334120 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.334136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.334835 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.334839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.334924 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.335394 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.335408 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.336862 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.338575 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.338579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.338581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.338788 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.339061 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.339065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.339067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.339151 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.339513 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.339516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.339618 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.339743 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.339892 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 20:12:35.340083 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.330761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.0.115.188:43657)
0924 20:12:35.331408 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.331413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.331429 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:12:35.331456 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.331458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.331516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.331597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.332294 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.333192 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.334276 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.334709 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.334815 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.337009 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.337013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.337015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.337038 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.338459 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.338885 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.338889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.338891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.338971 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.339247 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.339250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.339252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.339420 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.339521 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.339795 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.339906 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576}
I20260924 20:12:35.343721 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.333291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59434)
0924 20:12:35.333453 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.333467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.333544 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.333982 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.333985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.334141 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.334234 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.334241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.335583 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.335588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.336423 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.336434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.337035 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.337041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.337685 (+ 644us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.338392 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.338407 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.339882 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.342234 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.342239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.342241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.342463 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.342746 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.342748 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.342750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.342821 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.343187 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.343190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.343271 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.343375 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.343516 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":513}
I20260924 20:12:35.344003 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.333789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.0.115.189:37581)
0924 20:12:35.333930 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.333936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.333952 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:12:35.334074 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.334078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.334128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.334200 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.334949 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.335642 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.336465 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.336767 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.336868 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.340165 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.340170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.340174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.340205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.342008 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.342521 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.342524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.342526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.342616 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.342923 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.342927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.342928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.343161 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.343365 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.343718 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.343834 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:12:35.343750 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.333423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.0.115.188:43657)
0924 20:12:35.333767 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.333774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.333782 (+ 8us) server_negotiation.cc:407] Connection header received
0924 20:12:35.333806 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.333808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.333867 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.334322 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.334383 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.335464 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.336557 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.336936 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.337030 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.340078 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.340084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.340087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.340114 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.342120 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.342563 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.342566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.342567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.342643 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.342922 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.342925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.342926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.343092 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.343198 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.343431 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.343566 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260924 20:12:35.344388 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361: Bootstrap starting.
I20260924 20:12:35.344851 23241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } attempt: 3
I20260924 20:12:35.346932 23316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: Bootstrap starting.
I20260924 20:12:35.347388 23166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:35.348346 23166 log.cc:824] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:12:35 dist-test-slave-2pm7 krb5kdc[23085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790280755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.115.189@KRBTEST.COM for kudu/127.0.115.188@KRBTEST.COM
I20260924 20:12:35.349618 23316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:35.350447 23316 log.cc:824] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:35.351174 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361: No bootstrap required, opened a new log
I20260924 20:12:35.352766 23316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: No bootstrap required, opened a new log
I20260924 20:12:35.353565 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.345250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.115.188:43657 (local address 127.0.0.1:59444)
0924 20:12:35.345392 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 20:12:35.345406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:12:35.345476 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:12:35.345766 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:12:35.345769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:12:35.345935 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:12:35.346011 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.346015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.346730 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.346735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.347329 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:12:35.347338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.347840 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.347844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:12:35.347924 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.348433 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:12:35.348455 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:12:35.349946 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:12:35.351887 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.351890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.351892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.352164 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.352472 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:12:35.352475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:12:35.352477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:12:35.352568 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:12:35.353048 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:12:35.353051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:12:35.353163 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 20:12:35.353266 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 20:12:35.353408 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 20:12:35.353986 462 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:12:35.354254 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.345286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.0.115.188:43657)
0924 20:12:35.345595 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.345598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.345608 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:12:35.345635 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.345637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.345669 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.345749 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.346117 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.346625 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.347477 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.347730 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.347840 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.350072 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.350076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.350079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.350102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.351763 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.352279 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.352282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.352284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.352368 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.352686 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.352689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.352691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.352915 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.353004 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.353991 (+ 987us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.354115 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260924 20:12:35.355171 23140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3615bb6e4a114285bb5c27e58a731361"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33730
I20260924 20:12:35.355837 23166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.356081 23166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3615bb6e4a114285bb5c27e58a731361, State: Initialized, Role: FOLLOWER
I20260924 20:12:35.356429 23166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.357312 23241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: Bootstrap starting.
I20260924 20:12:35.357447 23316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.357684 23316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368936bfe9684a99a8d7a264865e15b0, State: Initialized, Role: FOLLOWER
I20260924 20:12:35.357861 23166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:35.357903 23140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 20:12:35.358008 23140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:35.357990 23316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.359251 23316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:35.359426 23140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.360162 23241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: Neither blocks nor log segments found. Creating new log.
I20260924 20:12:35.361394 23140 leader_election.cc:288] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [CANDIDATE]: Term 1 election: Requested vote from peers a3da11841d624f01a1bd51a9b1300d1f (127.0.115.189:37581), 368936bfe9684a99a8d7a264865e15b0 (127.0.115.188:43657)
I20260924 20:12:35.363554 23290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3615bb6e4a114285bb5c27e58a731361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "368936bfe9684a99a8d7a264865e15b0"
I20260924 20:12:35.363792 23290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [term 0 FOLLOWER]: Advancing to term 1
W20260924 20:12:35.364723 23347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:35.364806 23347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:35.365187 23290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3615bb6e4a114285bb5c27e58a731361 in term 1.
W20260924 20:12:35.365358 23336 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:35.365440 23336 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:35.365542 23325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.365638 23117 leader_election.cc:302] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3615bb6e4a114285bb5c27e58a731361, 368936bfe9684a99a8d7a264865e15b0; no voters:
I20260924 20:12:35.365803 23325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:35.365942 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.366083 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:35.366103 23357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:12:35.366245 23357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [term 1 LEADER]: Becoming Leader. State: Replica: 3615bb6e4a114285bb5c27e58a731361, State: Running, Role: LEADER
I20260924 20:12:35.366714 23357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.366948 23241 log.cc:824] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:12:35.369288 23241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: No bootstrap required, opened a new log
I20260924 20:12:35.371516 23098 master_runner.cc:186] Error getting master registration for 127.0.115.189:37581: OK, instance_id { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" instance_seqno: 1790280755089357 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:12:35.371539 23247 master_runner.cc:186] Error getting master registration for 127.0.115.189:37581: OK, instance_id { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" instance_seqno: 1790280755089357 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:12:35.374987 23247 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:12:35.375100 23098 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:12:35.375344 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3615bb6e4a114285bb5c27e58a731361. Latest consensus state: current_term: 1 leader_uuid: "3615bb6e4a114285bb5c27e58a731361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.375559 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:35.378504 23241 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.378767 23241 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3da11841d624f01a1bd51a9b1300d1f, State: Initialized, Role: FOLLOWER
I20260924 20:12:35.379112 23241 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } }
I20260924 20:12:35.380657 23241 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:12:35.380705 23215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3615bb6e4a114285bb5c27e58a731361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a3da11841d624f01a1bd51a9b1300d1f"
I20260924 20:12:35.380935 23215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:12:35.382459 23215 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3615bb6e4a114285bb5c27e58a731361 in term 1.
I20260924 20:12:35.389424 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.365189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.0.115.189:37581)
0924 20:12:35.365309 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.365313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.376652 (+ 11339us) server_negotiation.cc:407] Connection header received
0924 20:12:35.376777 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.376780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.376834 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.376917 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.378443 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.379176 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.383262 (+ 4086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.383566 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.383642 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.385496 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.385500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.385503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.385530 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.387418 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.387912 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.387915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.387917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.388046 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.388378 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.388382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.388386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.388645 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.388773 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.389124 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.389266 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:12:35.390069 23360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:12:35.391198 23360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 20:12:35.392947 23372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:12:35.393028 23372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:12:35.393325 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.393460 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:35.395737 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.363551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.0.115.188:43657)
0924 20:12:35.363670 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.363675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.372248 (+ 8573us) server_negotiation.cc:407] Connection header received
0924 20:12:35.372381 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.372385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.372435 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.372517 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.377005 (+ 4488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.377820 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.378951 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.379294 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.379375 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.381350 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.381354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.381356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.381387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.383361 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.392309 (+ 8948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.392314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.392315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.392425 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.393174 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.393177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.393179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.393423 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.393548 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.395437 (+ 1889us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.395570 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 20:12:35.397142 23290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [term 1 FOLLOWER]: Refusing update from remote peer 3615bb6e4a114285bb5c27e58a731361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:35.397833 23357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:35.398046 23215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [term 1 FOLLOWER]: Refusing update from remote peer 3615bb6e4a114285bb5c27e58a731361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:12:35.398556 23357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:12:35.398725 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:35.364401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.0.115.190:33171)
0924 20:12:35.364510 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.364515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.376273 (+ 11758us) server_negotiation.cc:407] Connection header received
0924 20:12:35.376815 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.376820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.376873 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.376925 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.377820 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.378572 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.381470 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:35.381869 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:35.381957 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:35.392795 (+ 10838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:12:35.392801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:12:35.392804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:12:35.392832 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:12:35.394800 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.397295 (+ 2495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.397299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.397302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.397397 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:12:35.397756 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:12:35.397761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:12:35.397765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:12:35.397982 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:12:35.398091 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:35.398443 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 20:12:35.398566 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:12:35.402464 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3615bb6e4a114285bb5c27e58a731361. Latest consensus state: current_term: 1 leader_uuid: "3615bb6e4a114285bb5c27e58a731361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.402616 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:35.402993 23325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3615bb6e4a114285bb5c27e58a731361. Latest consensus state: current_term: 1 leader_uuid: "3615bb6e4a114285bb5c27e58a731361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.403131 23325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:35.406999 23357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3615bb6e4a114285bb5c27e58a731361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.407150 23357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:35.408437 23325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3615bb6e4a114285bb5c27e58a731361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.408557 23325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:35.408960 23374 mvcc.cc:205] Tried to move back new op lower bound from 7332989974099623936 to 7332989973985730560. Current Snapshot: MvccSnapshot[applied={T|T < 7332989974099623936}]
I20260924 20:12:35.410104 23375 mvcc.cc:205] Tried to move back new op lower bound from 7332989974099623936 to 7332989973985730560. Current Snapshot: MvccSnapshot[applied={T|T < 7332989974099623936}]
I20260924 20:12:35.412129 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3615bb6e4a114285bb5c27e58a731361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.412271 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361 [sys.catalog]: This master's current role is: LEADER
I20260924 20:12:35.413496 23361 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3615bb6e4a114285bb5c27e58a731361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3615bb6e4a114285bb5c27e58a731361" member_type: VOTER last_known_addr { host: "127.0.115.190" port: 33171 } } peers { permanent_uuid: "a3da11841d624f01a1bd51a9b1300d1f" member_type: VOTER last_known_addr { host: "127.0.115.189" port: 37581 } } peers { permanent_uuid: "368936bfe9684a99a8d7a264865e15b0" member_type: VOTER last_known_addr { host: "127.0.115.188" port: 43657 } } }
I20260924 20:12:35.413618 23361 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:12:35.414427 23376 mvcc.cc:205] Tried to move back new op lower bound from 7332989974099623936 to 7332989973985730560. Current Snapshot: MvccSnapshot[applied={T|T < 7332989974099623936}]
I20260924 20:12:35.417874 23360 catalog_manager.cc:1347] Generated new cluster ID: 812a66a67f9b4f5c99ca78ec7438869f
I20260924 20:12:35.417953 23360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:12:35.430540 23360 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:12:35.431878 23360 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:12:35.438891 23360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3615bb6e4a114285bb5c27e58a731361: Generated new TSK 0
I20260924 20:12:35.440351 23360 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:12:35.472565 23319 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.470891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.0.115.188:43657)
0924 20:12:35.471044 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.471048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.471415 (+ 367us) server_negotiation.cc:407] Connection header received
0924 20:12:35.471615 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.471619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.471668 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.471723 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.472414 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59460: BlockingRecv error: recv got EOF from 127.0.0.1:59460 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 20:12:35.475169 23169 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.473402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.0.115.190:33171)
0924 20:12:35.473524 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.473530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.474198 (+ 668us) server_negotiation.cc:407] Connection header received
0924 20:12:35.474396 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.474400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.474451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.474520 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.475031 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33744: BlockingRecv error: recv got EOF from 127.0.0.1:33744 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 20:12:35.476845 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.473935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.0.115.189:37581)
0924 20:12:35.474045 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.474049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.475916 (+ 1867us) server_negotiation.cc:407] Connection header received
0924 20:12:35.476121 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.476124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.476174 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.476235 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.476710 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48602: BlockingRecv error: recv got EOF from 127.0.0.1:48602 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:12:35.481346 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.479681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.0.115.189:37581)
0924 20:12:35.479790 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.479794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.480356 (+ 562us) server_negotiation.cc:407] Connection header received
0924 20:12:35.480584 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.480587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.480671 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.480726 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.481238 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48608: BlockingRecv error: recv got EOF from 127.0.0.1:48608 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:12:35.497490 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.488715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.115.189:37581)
0924 20:12:35.488871 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.488875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.496388 (+ 7513us) server_negotiation.cc:407] Connection header received
0924 20:12:35.496633 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.496637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.496699 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.496765 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.497360 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48614: BlockingRecv error: recv got EOF from 127.0.0.1:48614 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 20:12:35.505380 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.501833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.0.115.189:37581)
0924 20:12:35.501952 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.501956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.504345 (+ 2389us) server_negotiation.cc:407] Connection header received
0924 20:12:35.504555 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.504558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.504613 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.504668 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.505255 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48626: BlockingRecv error: recv got EOF from 127.0.0.1:48626 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 20:12:35.514205 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.512074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.0.115.189:37581)
0924 20:12:35.512246 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.512251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.512978 (+ 727us) server_negotiation.cc:407] Connection header received
0924 20:12:35.513213 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.513217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.513361 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.513448 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.514056 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48642: BlockingRecv error: recv got EOF from 127.0.0.1:48642 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 20:12:35.526998 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.525126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.0.115.189:37581)
0924 20:12:35.525260 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.525265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.525895 (+ 630us) server_negotiation.cc:407] Connection header received
0924 20:12:35.526148 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.526152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.526208 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.526290 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.526876 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48644: BlockingRecv error: recv got EOF from 127.0.0.1:48644 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 20:12:35.548103 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.546143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.0.115.189:37581)
0924 20:12:35.546276 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.546281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.546921 (+ 640us) server_negotiation.cc:407] Connection header received
0924 20:12:35.547182 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.547186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.547248 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.547306 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.547918 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48658: BlockingRecv error: recv got EOF from 127.0.0.1:48658 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 20:12:35.584868 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.582973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.0.115.189:37581)
0924 20:12:35.583115 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.583120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.583727 (+ 607us) server_negotiation.cc:407] Connection header received
0924 20:12:35.584028 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.584033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.584107 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.584165 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.584747 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48664: BlockingRecv error: recv got EOF from 127.0.0.1:48664 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:12:35.652947 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.651737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.0.115.189:37581)
0924 20:12:35.651894 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.651898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.652033 (+ 135us) server_negotiation.cc:407] Connection header received
0924 20:12:35.652232 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.652235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.652283 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.652374 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.652843 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48668: BlockingRecv error: recv got EOF from 127.0.0.1:48668 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 20:12:35.784523 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:35.783212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.0.115.189:37581)
0924 20:12:35.783401 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:35.783406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:35.783545 (+ 139us) server_negotiation.cc:407] Connection header received
0924 20:12:35.783726 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:35.783730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:35.783788 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:35.783875 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:35.784378 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48680: BlockingRecv error: recv got EOF from 127.0.0.1:48680 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 20:12:36.043835 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:12:36.042676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.115.189:37581)
0924 20:12:36.042853 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:36.042859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:36.042877 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:12:36.043071 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:36.043075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:36.043130 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:36.043221 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:12:36.043703 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48684: BlockingRecv error: recv got EOF from 127.0.0.1:48684 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 20:12:36.366196 23347 catalog_manager.cc:1259] Loaded cluster ID: 812a66a67f9b4f5c99ca78ec7438869f
I20260924 20:12:36.366281 23347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: loading cluster ID for follower catalog manager: success
I20260924 20:12:36.368003 23347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: acquiring CA information for follower catalog manager: success
I20260924 20:12:36.369195 23347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 368936bfe9684a99a8d7a264865e15b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:12:36.394443 23372 catalog_manager.cc:1259] Loaded cluster ID: 812a66a67f9b4f5c99ca78ec7438869f
I20260924 20:12:36.394548 23372 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: loading cluster ID for follower catalog manager: success
I20260924 20:12:36.396379 23372 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: acquiring CA information for follower catalog manager: success
I20260924 20:12:36.397655 23372 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3da11841d624f01a1bd51a9b1300d1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:12:36.562568 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:36.558179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.0.115.189:37581)
0924 20:12:36.558440 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:36.558444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:36.558615 (+ 171us) server_negotiation.cc:407] Connection header received
0924 20:12:36.558724 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:36.558727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:36.558858 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:36.558970 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:36.559282 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.560032 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.561083 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.561373 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.561465 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:36.561768 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:36.562239 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:36.562337 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:36.562389 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 20:12:36.562441 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260924 20:12:36.569394 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:36.565666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.0.115.188:43657)
0924 20:12:36.565916 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:36.565923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:36.565937 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:12:36.566033 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:36.566035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:36.566150 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:36.566274 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:36.566493 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.567020 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.567847 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.568136 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.568251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:36.568552 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:36.568981 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:36.569141 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:36.569182 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 20:12:36.569237 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":57,"threads_started":1}
I20260924 20:12:36.576843 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:36.573056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.0.115.189:37581)
0924 20:12:36.573197 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:36.573200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:36.573210 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:12:36.573251 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:36.573253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:36.573293 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:36.573376 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:36.573650 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.574179 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.575556 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.575791 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.575848 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:36.576320 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:36.576407 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:36.576486 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:36.576613 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 20:12:36.576681 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:12:36.576843 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:36.572615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.0.115.188:43657)
0924 20:12:36.572741 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:36.572746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:36.572982 (+ 236us) server_negotiation.cc:407] Connection header received
0924 20:12:36.573150 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:36.573153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:36.573195 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:36.573239 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:36.573583 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.574376 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.575542 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.575855 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.575976 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:36.576349 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:36.576413 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:36.576459 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:36.576672 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 20:12:36.576720 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:12:36.577860 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:12:36.572488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.0.115.190:33171)
0924 20:12:36.572782 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 20:12:36.572787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:12:36.572980 (+ 193us) server_negotiation.cc:407] Connection header received
0924 20:12:36.573098 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:12:36.573104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:12:36.573275 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:12:36.573403 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:12:36.573753 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.574542 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.575491 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:12:36.575849 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:12:36.575976 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:12:36.576425 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:12:36.576809 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:12:36.576901 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:12:36.577655 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 20:12:36.577723 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
I20260924 20:12:36.581116 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 23098
I20260924 20:12:36.588084 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 23173
I20260924 20:12:36.594022 462 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIXC3Dg/build/debug/bin/kudu with pid 23247
2026-09-24T20:12:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3085 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6208 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (290052 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST